Jul 14 11:35:40.589236 ip-10-0-132-116 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 11:35:40.589245 ip-10-0-132-116 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 11:35:40.589252 ip-10-0-132-116 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 11:35:40.589454 ip-10-0-132-116 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 11:35:51.751271 ip-10-0-132-116 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 11:35:51.751286 ip-10-0-132-116 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7a42d80c964c44ecba2838558634c9eb -- Jul 14 11:37:46.809355 ip-10-0-132-116 systemd[1]: Starting Kubernetes Kubelet... Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.201259 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206317 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206338 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206341 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206345 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206348 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206353 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206357 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 11:37:47.306944 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206361 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 11:37:47.265963 ip-10-0-132-116 systemd[1]: Started Kubernetes Kubelet. Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206364 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206367 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206370 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206372 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206375 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206377 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206380 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206384 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206387 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206391 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206393 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206396 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206399 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206402 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206404 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206407 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206410 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206413 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 11:37:47.307999 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206417 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206420 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206423 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206426 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206430 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206432 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206435 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206438 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206440 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206443 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206446 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206448 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206450 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206453 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206455 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206458 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206460 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206463 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206465 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206468 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 11:37:47.308729 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206471 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206473 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206476 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206479 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206481 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206484 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206486 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206489 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206491 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206493 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206497 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206499 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206502 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206504 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206508 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206510 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206513 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206515 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206518 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206521 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 11:37:47.310640 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206524 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206526 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206529 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206533 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206536 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206538 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206541 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206543 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206546 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206548 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206551 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206553 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206556 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206559 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206561 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206564 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206566 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206569 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206571 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 11:37:47.315167 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206574 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206576 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206578 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206581 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206583 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206586 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206589 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206592 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206595 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206597 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206600 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206602 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206605 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.206608 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208606 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208617 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208621 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208624 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208628 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208631 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 11:37:47.320024 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208633 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208636 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208639 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208641 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208644 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208647 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208649 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208652 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208654 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208657 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208661 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208666 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208669 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208672 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208676 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208679 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208682 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208685 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208688 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 11:37:47.321681 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208690 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208693 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208696 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208699 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208701 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208704 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208706 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208710 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208714 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208718 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208720 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208723 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208727 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208730 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208732 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208735 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208737 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208740 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208743 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208745 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 11:37:47.323767 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208748 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208751 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208753 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208755 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208758 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208760 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208763 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208765 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208768 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208770 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208773 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208775 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208778 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208780 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208782 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208785 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208787 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208790 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208793 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 11:37:47.325367 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208796 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208800 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208802 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208805 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208808 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208810 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208812 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208815 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208817 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208820 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208822 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208825 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208828 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208830 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208833 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208835 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208837 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208840 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208842 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208846 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 11:37:47.326237 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208848 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208851 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208853 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208855 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208858 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208860 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208863 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208865 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208867 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208870 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208872 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208875 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208877 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208880 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.208882 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209264 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209271 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209275 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209278 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209281 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 11:37:47.327027 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209284 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209287 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209290 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209292 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209295 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209297 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209299 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209302 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209304 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209307 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209310 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209312 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209315 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209317 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209320 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209322 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209325 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209327 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209331 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209333 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 11:37:47.327655 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209336 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209339 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209341 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209343 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209346 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209348 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209351 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209356 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209358 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209361 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209365 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209369 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209372 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209375 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209377 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209380 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209382 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209387 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209390 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 11:37:47.328465 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209393 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209396 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209399 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209401 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209404 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209408 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209410 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209413 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209415 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209418 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209420 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209423 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209425 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209427 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209430 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209432 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209435 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209437 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209440 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209442 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 11:37:47.329070 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209446 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209448 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209451 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209453 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209457 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209460 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209462 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209465 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209467 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209470 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209473 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209475 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209477 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209480 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209482 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209485 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209487 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209489 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209493 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 11:37:47.329888 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209495 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209497 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209500 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209504 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209508 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209511 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209514 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209516 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209519 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209522 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209524 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209526 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209529 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209532 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209535 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.209537 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209601 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209608 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209615 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209621 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209627 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 11:37:47.330508 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209632 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209638 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209643 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209648 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209652 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209658 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209662 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209664 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209668 2562 flags.go:64] FLAG: --cgroup-root="" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209670 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209673 2562 flags.go:64] FLAG: --client-ca-file="" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209677 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209679 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209684 2562 flags.go:64] FLAG: --cluster-domain="" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209687 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209690 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209693 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209697 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209707 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209710 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209713 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209717 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209720 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209724 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 11:37:47.331171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209728 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209731 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209734 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209738 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209741 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209743 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209746 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209749 2562 flags.go:64] FLAG: --enable-server="true" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209752 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209755 2562 flags.go:64] FLAG: --event-burst="100" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209759 2562 flags.go:64] FLAG: --event-qps="50" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209761 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209765 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209767 2562 flags.go:64] FLAG: --eviction-hard="" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209771 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209773 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209776 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209779 2562 flags.go:64] FLAG: --eviction-soft="" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209782 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209785 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209788 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209791 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209794 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209797 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209799 2562 flags.go:64] FLAG: --feature-gates="" Jul 14 11:37:47.332176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209803 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209806 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209808 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209811 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209814 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209817 2562 flags.go:64] FLAG: --help="false" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209821 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209826 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209829 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209832 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209835 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209838 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209841 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209843 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209846 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209849 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209852 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209855 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209858 2562 flags.go:64] FLAG: --kube-reserved="" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209861 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209863 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209866 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209869 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209871 2562 flags.go:64] FLAG: --lock-file="" Jul 14 11:37:47.333384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209874 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209877 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209879 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209885 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209888 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209891 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209894 2562 flags.go:64] FLAG: --logging-format="text" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209896 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209899 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209902 2562 flags.go:64] FLAG: --manifest-url="" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209905 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209908 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209911 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209915 2562 flags.go:64] FLAG: --max-pods="110" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209919 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209924 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209927 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209929 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209932 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209935 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209938 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209945 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209947 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209950 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 11:37:47.334632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209953 2562 flags.go:64] FLAG: --pod-cidr="" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209956 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209961 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209963 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209966 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209969 2562 flags.go:64] FLAG: --port="10250" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209972 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209975 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0811adeec1203a171" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209978 2562 flags.go:64] FLAG: --qos-reserved="" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209980 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209983 2562 flags.go:64] FLAG: --register-node="true" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209986 2562 flags.go:64] FLAG: --register-with-taints="" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209989 2562 flags.go:64] FLAG: --registry-burst="10" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209992 2562 flags.go:64] FLAG: --registry-qps="5" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209995 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.209998 2562 flags.go:64] FLAG: --reserved-memory="" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210001 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210004 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210007 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210010 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210012 2562 flags.go:64] FLAG: --runonce="false" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210015 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210019 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210022 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210026 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 11:37:47.338094 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210029 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210032 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210035 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210038 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210040 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210043 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210046 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210049 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210052 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210055 2562 flags.go:64] FLAG: --system-cgroups="" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210058 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210063 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210065 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210068 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210072 2562 flags.go:64] FLAG: --tls-min-version="" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210086 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210089 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210092 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210095 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210097 2562 flags.go:64] FLAG: --v="2" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210101 2562 flags.go:64] FLAG: --version="false" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210105 2562 flags.go:64] FLAG: --vmodule="" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210109 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.210112 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210225 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 11:37:47.339609 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210228 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210232 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210234 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210237 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210243 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210246 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210249 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210252 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210254 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210257 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210260 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210263 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210265 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210267 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210270 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210272 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210275 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210277 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210280 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210282 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 11:37:47.340604 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210285 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210287 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210290 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210292 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210294 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210297 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210299 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210303 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210306 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210312 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210315 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210317 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210320 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210322 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210325 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210327 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210331 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210334 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210337 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210340 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 11:37:47.341736 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210342 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210345 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210347 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210349 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210352 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210354 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210357 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210359 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210362 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210364 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210366 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210369 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210372 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210374 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210377 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210379 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210382 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210384 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210386 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210389 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 11:37:47.342688 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210391 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210393 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210396 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210399 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210401 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210403 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210406 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210410 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210414 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210417 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210422 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210424 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210427 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210429 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210432 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210434 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210437 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210440 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210442 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210445 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 11:37:47.343734 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210447 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210450 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210452 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210455 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210457 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210460 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210462 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210465 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210467 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210470 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210472 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210474 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210477 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210481 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210484 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210490 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210492 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 11:37:47.344292 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.210495 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.211909 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.217790 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.217805 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.217820 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.217825 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217878 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217883 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217886 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217889 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217891 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217894 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217897 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217899 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217902 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 11:37:47.478396 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217904 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217907 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217909 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217912 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217914 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217916 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217919 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217922 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217924 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217927 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217929 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217932 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217934 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217936 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217939 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217942 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217944 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217948 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217951 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217954 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 11:37:47.479120 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217959 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217961 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217964 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217969 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217972 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217974 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217977 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217979 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217983 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217986 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217989 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217991 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217994 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217996 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.217999 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218001 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218004 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218007 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218010 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218012 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 11:37:47.480344 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218015 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218017 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218019 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218022 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218024 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218027 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218029 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218032 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218034 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218036 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218039 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218043 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218047 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218051 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218053 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218057 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218060 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218062 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218064 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218067 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 11:37:47.482818 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218069 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218072 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218086 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218090 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218092 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218095 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218097 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218100 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218102 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218105 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218108 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218110 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218113 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218115 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218118 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218120 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218123 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218125 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218128 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 11:37:47.483395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218130 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218132 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218135 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218137 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218140 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218143 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218146 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218151 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218155 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218158 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218161 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.218165 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218267 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218271 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218274 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 11:37:47.483894 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218277 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218279 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218282 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218284 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218287 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218289 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218292 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218294 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218297 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218300 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218302 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218304 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218308 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218312 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218314 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218317 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218320 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218322 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218325 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 11:37:47.484395 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218328 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218330 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218333 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218337 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218339 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218342 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218345 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218349 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218352 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218354 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218357 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218359 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218362 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218364 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218366 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218369 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218371 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218374 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218376 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218379 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 11:37:47.484905 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218381 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218384 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218386 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218389 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218391 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218394 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218396 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218399 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218401 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218404 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218406 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218408 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218411 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218413 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218416 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218420 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218423 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218426 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218428 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218433 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 11:37:47.485459 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218436 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218438 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218441 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218443 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218446 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218448 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218451 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218453 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218456 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218458 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218461 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218463 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218465 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218468 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218470 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218473 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218475 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218478 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218480 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 11:37:47.486117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218483 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218486 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218490 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218492 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218495 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218498 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218501 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218505 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218508 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218511 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218513 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218516 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218519 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218522 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218525 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218527 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218530 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 11:37:47.486650 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:47.218533 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.218538 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.218627 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.220467 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.221922 2562 server.go:1019] "Starting client certificate rotation" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.222021 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.222052 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.224172 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.227605 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.237762 2562 log.go:25] "Validated CRI v1 runtime API" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.239857 2562 log.go:25] "Validated CRI v1 image API" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.240496 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 11:37:47.487204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.242512 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a71d6999-c6bf-4e1c-86e0-87f4f914dadf:/dev/nvme0n1p3 eac68ae7-a897-40bc-be0b-69dc3c74d1de:/dev/nvme0n1p4] Jul 14 11:37:47.487570 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.242527 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 11:37:47.487730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.248101 2562 manager.go:216] Machine: {Timestamp:2026-07-14 11:37:47.246745493 +0000 UTC m=+0.322600072 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200796 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2632fa04e94e28d5518ad09c81555b SystemUUID:ec2632fa-04e9-4e28-d551-8ad09c81555b BootID:7a42d80c-964c-44ec-ba28-38558634c9eb Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:08:a7:86:29:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:08:a7:86:29:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:04:51:13:f5:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 11:37:47.487730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.248196 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 11:37:47.487730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.248260 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 11:37:47.487730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.250883 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 11:37:47.487730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.250904 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-116.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 11:37:47.487730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.251556 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 11:37:47.487730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.251564 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 11:37:47.487730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.251571 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.252150 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.252189 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.253211 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.256257 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.256356 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.256381 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.256389 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.256401 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.256413 2562 kubelet.go:398] "Adding apiserver pod source" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.256422 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.256698 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.256712 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.258755 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.260363 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.260388 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.260987 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261000 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261005 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261010 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261018 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261023 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261170 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261175 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261181 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261186 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261196 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261203 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261767 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.261773 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.264614 2562 server.go:1268] "Started kubelet" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.264699 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.264782 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.264703 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.266060 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.270441 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-116.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.270469 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.270639 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.270697 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 11:37:47.488019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.272365 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.273429 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.273441 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.273616 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274032 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274192 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274205 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274321 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274332 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274366 2562 factory.go:55] Registering systemd factory Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274382 2562 factory.go:223] Registration of the systemd container factory successfully Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274602 2562 factory.go:153] Registering CRI-O factory Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274614 2562 factory.go:223] Registration of the crio container factory successfully Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274666 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274690 2562 factory.go:103] Registering Raw factory Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.274704 2562 manager.go:1196] Started watching for new ooms in manager Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.275184 2562 manager.go:349] Starting recovery of all containers Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.276453 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.276471 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.276553 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-116.ec2.internal.18c224fc0746fdb9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-116.ec2.internal,UID:ip-10-0-132-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-116.ec2.internal,},FirstTimestamp:2026-07-14 11:37:47.264589241 +0000 UTC m=+0.340443819,LastTimestamp:2026-07-14 11:37:47.264589241 +0000 UTC m=+0.340443819,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-116.ec2.internal,}" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.282187 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fdwsr" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.286108 2562 manager.go:354] Recovery completed Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.290035 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.290365 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fdwsr" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.291721 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientMemory" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.291746 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.291758 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientPID" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.292255 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.292263 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.292285 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.292327 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.293042 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientMemory" Jul 14 11:37:47.488975 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.293064 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.293089 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientPID" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.293433 2562 policy_none.go:49] "None policy: Start" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.293449 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.293461 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.294269 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-116.ec2.internal.18c224fc08e52439 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-116.ec2.internal,UID:ip-10-0-132-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-116.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-116.ec2.internal,},FirstTimestamp:2026-07-14 11:37:47.291731001 +0000 UTC m=+0.367585580,LastTimestamp:2026-07-14 11:37:47.291731001 +0000 UTC m=+0.367585580,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-116.ec2.internal,}" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.294725 2562 policy_none.go:47] "Start" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.349632 2562 manager.go:337] "Starting Device Plugin manager" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.349751 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.349762 2562 server.go:87] "Starting device plugin registration server" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.349969 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.349980 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.350069 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.350171 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.350180 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.350590 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.350635 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.450281 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.451041 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientMemory" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.451062 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.451072 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientPID" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.451111 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.471368 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.471384 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-116.ec2.internal\": node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.471944 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.473010 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.473050 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.473069 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.473119 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.475318 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.504523 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.574193 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal"] Jul 14 11:37:47.629982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.574242 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.574918 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientMemory" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.574940 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.574948 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientPID" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20e26076d2b186e5e7dec3f4e5d6254b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal\" (UID: \"20e26076d2b186e5e7dec3f4e5d6254b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20e26076d2b186e5e7dec3f4e5d6254b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal\" (UID: \"20e26076d2b186e5e7dec3f4e5d6254b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576193 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576295 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576322 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576819 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientMemory" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576819 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientMemory" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576858 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576867 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientPID" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576842 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.576922 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientPID" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.578212 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.578241 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.578841 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientMemory" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.578867 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.578882 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeHasSufficientPID" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.605197 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.609549 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-116.ec2.internal\" not found" node="ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.631152 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.615689 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-116.ec2.internal\" not found" node="ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.797838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.676606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20e26076d2b186e5e7dec3f4e5d6254b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal\" (UID: \"20e26076d2b186e5e7dec3f4e5d6254b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.797838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.676629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20e26076d2b186e5e7dec3f4e5d6254b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal\" (UID: \"20e26076d2b186e5e7dec3f4e5d6254b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.797838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.676650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef29f282130d13891da160528c132f5f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-116.ec2.internal\" (UID: \"ef29f282130d13891da160528c132f5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.797838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.676720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20e26076d2b186e5e7dec3f4e5d6254b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal\" (UID: \"20e26076d2b186e5e7dec3f4e5d6254b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.797838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.676803 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20e26076d2b186e5e7dec3f4e5d6254b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal\" (UID: \"20e26076d2b186e5e7dec3f4e5d6254b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.797838 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.706279 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:47.797838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.777469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef29f282130d13891da160528c132f5f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-116.ec2.internal\" (UID: \"ef29f282130d13891da160528c132f5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.797838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.777509 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef29f282130d13891da160528c132f5f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-116.ec2.internal\" (UID: \"ef29f282130d13891da160528c132f5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.966579 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.806675 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:47.966579 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:47.908697 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:47.966579 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.912045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" Jul 14 11:37:47.966579 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:47.918064 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal" Jul 14 11:37:48.135643 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:48.010189 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:48.135643 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:48.112471 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:48.287505 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:48.214073 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:48.287505 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.221748 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 11:37:48.287505 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.221905 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 11:37:48.443433 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.294635 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 11:32:47 +0000 UTC" deadline="2027-12-24 06:52:44.201231286 +0000 UTC" Jul 14 11:37:48.443433 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.294657 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12667h14m55.906577321s" Jul 14 11:37:48.443433 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:48.314453 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:48.443433 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:48.415054 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:48.607834 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:48.515476 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:48.607834 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.525239 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 11:37:48.607834 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:48.541804 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e26076d2b186e5e7dec3f4e5d6254b.slice/crio-48c57b8a2ecc0011563d0718760cd11c0c2a94d6e45f4ad9aad67c60b41c25f2 WatchSource:0}: Error finding container 48c57b8a2ecc0011563d0718760cd11c0c2a94d6e45f4ad9aad67c60b41c25f2: Status 404 returned error can't find the container with id 48c57b8a2ecc0011563d0718760cd11c0c2a94d6e45f4ad9aad67c60b41c25f2 Jul 14 11:37:48.607834 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:48.542352 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef29f282130d13891da160528c132f5f.slice/crio-9b4212ef5e457013b9c37fd099ed049008f63a46d730a72fe68a9389e4694320 WatchSource:0}: Error finding container 9b4212ef5e457013b9c37fd099ed049008f63a46d730a72fe68a9389e4694320: Status 404 returned error can't find the container with id 9b4212ef5e457013b9c37fd099ed049008f63a46d730a72fe68a9389e4694320 Jul 14 11:37:48.607834 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.546597 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 11:37:48.766973 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:48.618106 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:48.766973 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.663140 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 11:37:48.766973 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:48.719452 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-116.ec2.internal\" not found" Jul 14 11:37:48.886873 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.783697 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 11:37:48.886873 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.873860 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" Jul 14 11:37:48.886873 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.886499 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 11:37:48.887803 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.887686 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal" Jul 14 11:37:48.899547 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:48.899520 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 11:37:49.258305 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.258222 2562 apiserver.go:52] "Watching apiserver" Jul 14 11:37:49.273188 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.273124 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 11:37:49.274281 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.274247 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-md69v","kube-system/global-pull-secret-syncer-xkhkk","kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt","openshift-cluster-node-tuning-operator/tuned-c562x","openshift-multus/multus-additional-cni-plugins-zx9tj","openshift-multus/multus-fjzx5","openshift-multus/network-metrics-daemon-t2c79","openshift-network-diagnostics/network-check-target-js79l","kube-system/konnectivity-agent-tmggf","openshift-dns/node-resolver-tfd9j","openshift-image-registry/node-ca-dgl7z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal","openshift-network-operator/iptables-alerter-gbmbj"] Jul 14 11:37:49.274517 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.274492 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 11:37:49.276288 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.276267 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.277723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.277698 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.279171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.279136 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.279684 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.279665 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 11:37:49.280026 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.279986 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ngbcl\"" Jul 14 11:37:49.280124 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.280048 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 11:37:49.280124 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.280105 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 11:37:49.280243 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.280210 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 11:37:49.280472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.280452 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 11:37:49.280548 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.280495 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.281146 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.281110 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-46h8g\"" Jul 14 11:37:49.282221 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.282200 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 11:37:49.282552 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.282534 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 11:37:49.282785 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.282768 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 11:37:49.283425 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.283390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.284222 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.284201 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 11:37:49.284369 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.284350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 11:37:49.284496 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.284471 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wm6ph\"" Jul 14 11:37:49.284698 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.284682 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 11:37:49.285473 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.285007 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 11:37:49.285473 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.285069 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.285473 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.285433 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xtwcs\"" Jul 14 11:37:49.287802 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.287388 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:49.287802 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.287466 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:37:49.287928 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.287911 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 11:37:49.287980 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.287915 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4bbp5\"" Jul 14 11:37:49.288034 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.288002 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 11:37:49.288566 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.288262 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 11:37:49.288566 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.288275 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 11:37:49.288566 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.288263 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zgzrx\"" Jul 14 11:37:49.288769 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.288642 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:49.288847 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.288814 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:37:49.290177 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.290151 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 11:37:49.290439 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.290397 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:49.290525 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.290483 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:37:49.292440 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.292261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:37:49.294763 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.294743 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.296053 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.296033 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.299907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.299570 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 11:37:49.299907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.299631 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 11:37:49.299907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.299678 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 11:37:49.299907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.299690 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 11:37:49.299907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.299579 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 11:37:49.299907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.299633 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 11:37:49.299907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.299571 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8xk6f\"" Jul 14 11:37:49.299907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.299857 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 11:37:49.300398 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.300379 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 11:37:49.300690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.300666 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4wb2t\"" Jul 14 11:37:49.301400 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.300971 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxr9q\"" Jul 14 11:37:49.301400 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.300986 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 11:37:49.301400 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.300990 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 11:37:49.301400 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.301204 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 11:37:49.317265 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.317241 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vnpfx" Jul 14 11:37:49.327412 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.327392 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vnpfx" Jul 14 11:37:49.375971 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.375944 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 11:37:49.385875 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.385851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-cnibin\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.385996 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.385882 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-hostroot\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.385996 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.385898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eed3a9e7-5412-4606-b01a-a852629061a5-hosts-file\") pod \"node-resolver-tfd9j\" (UID: \"eed3a9e7-5412-4606-b01a-a852629061a5\") " pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.385996 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.385919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-kubernetes\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.385996 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.385947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:49.385996 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.385972 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-socket-dir-parent\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.386279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhjg2\" (UniqueName: \"kubernetes.io/projected/fc95de41-f270-4b20-b65a-57b877df87d7-kube-api-access-mhjg2\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.386279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-kubelet\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.386279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386127 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c881463-b72f-4fe7-a835-744f68a507f9-host-slash\") pod \"iptables-alerter-gbmbj\" (UID: \"3c881463-b72f-4fe7-a835-744f68a507f9\") " pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.386279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-host\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.386279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-system-cni-dir\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.386279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-run-ovn\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.386279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386233 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rst\" (UniqueName: \"kubernetes.io/projected/3c881463-b72f-4fe7-a835-744f68a507f9-kube-api-access-p6rst\") pod \"iptables-alerter-gbmbj\" (UID: \"3c881463-b72f-4fe7-a835-744f68a507f9\") " pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.386279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386257 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-run\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.386279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-sys\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386303 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p27mf\" (UniqueName: \"kubernetes.io/projected/18eb4773-aa04-4688-9217-2d6bf662d310-kube-api-access-p27mf\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-run-openvswitch\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jhd\" (UniqueName: \"kubernetes.io/projected/f7eac93e-56a6-4d90-aa2e-507659c30532-kube-api-access-48jhd\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ff938fe-df60-4fb5-b558-e7832e86ff4b-host\") pod \"node-ca-dgl7z\" (UID: \"1ff938fe-df60-4fb5-b558-e7832e86ff4b\") " pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc95de41-f270-4b20-b65a-57b877df87d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-registration-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386486 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-sys-fs\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386511 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vjw\" (UniqueName: \"kubernetes.io/projected/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-kube-api-access-h5vjw\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-sysctl-conf\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-lib-modules\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386588 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-var-lib-cni-bin\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386623 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-run-multus-certs\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.386714 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8832e47-e0b9-43bc-ba93-3da7fef69778-cni-binary-copy\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-cnibin\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386774 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/55ed8677-a1e3-4247-8b7f-0cca2d6cd535-agent-certs\") pod \"konnectivity-agent-tmggf\" (UID: \"55ed8677-a1e3-4247-8b7f-0cca2d6cd535\") " pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-etc-openvswitch\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7eac93e-56a6-4d90-aa2e-507659c30532-ovn-node-metrics-cert\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386846 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7eac93e-56a6-4d90-aa2e-507659c30532-ovnkube-script-lib\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386871 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqjq\" (UniqueName: \"kubernetes.io/projected/eed3a9e7-5412-4606-b01a-a852629061a5-kube-api-access-mdqjq\") pod \"node-resolver-tfd9j\" (UID: \"eed3a9e7-5412-4606-b01a-a852629061a5\") " pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386896 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6gg5\" (UniqueName: \"kubernetes.io/projected/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-kube-api-access-x6gg5\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-etc-kubernetes\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc95de41-f270-4b20-b65a-57b877df87d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386965 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-run-netns\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.386989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-modprobe-d\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.387388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-os-release\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-dbus\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-systemd-units\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-sysconfig\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387073 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-sysctl-d\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-var-lib-kubelet\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-node-log\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-var-lib-cni-multus\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-run-ovn-kubernetes\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387223 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-cni-netd\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387248 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/55ed8677-a1e3-4247-8b7f-0cca2d6cd535-konnectivity-ca\") pod \"konnectivity-agent-tmggf\" (UID: \"55ed8677-a1e3-4247-8b7f-0cca2d6cd535\") " pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-run-systemd\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-log-socket\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.387953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387391 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8m8z\" (UniqueName: \"kubernetes.io/projected/b8832e47-e0b9-43bc-ba93-3da7fef69778-kube-api-access-m8m8z\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387494 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-var-lib-openvswitch\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-os-release\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387610 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-system-cni-dir\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387633 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ff938fe-df60-4fb5-b558-e7832e86ff4b-serviceca\") pod \"node-ca-dgl7z\" (UID: \"1ff938fe-df60-4fb5-b558-e7832e86ff4b\") " pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387655 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w79p2\" (UniqueName: \"kubernetes.io/projected/1ff938fe-df60-4fb5-b558-e7832e86ff4b-kube-api-access-w79p2\") pod \"node-ca-dgl7z\" (UID: \"1ff938fe-df60-4fb5-b558-e7832e86ff4b\") " pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c881463-b72f-4fe7-a835-744f68a507f9-iptables-alerter-script\") pod \"iptables-alerter-gbmbj\" (UID: \"3c881463-b72f-4fe7-a835-744f68a507f9\") " pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-run-netns\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387728 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-conf-dir\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-slash\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387787 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7eac93e-56a6-4d90-aa2e-507659c30532-ovnkube-config\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.388620 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-socket-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-cni-dir\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387867 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc95de41-f270-4b20-b65a-57b877df87d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-kubelet-config\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-cni-bin\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed3a9e7-5412-4606-b01a-a852629061a5-tmp-dir\") pod \"node-resolver-tfd9j\" (UID: \"eed3a9e7-5412-4606-b01a-a852629061a5\") " pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.387976 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-systemd\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.388001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-run-k8s-cni-cncf-io\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.388039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-var-lib-kubelet\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.388074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-daemon-config\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.388119 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.388144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7eac93e-56a6-4d90-aa2e-507659c30532-env-overrides\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.388195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-device-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.388224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18eb4773-aa04-4688-9217-2d6bf662d310-etc-tuned\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.389472 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.388246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18eb4773-aa04-4688-9217-2d6bf662d310-tmp\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.477957 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.477883 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" event={"ID":"20e26076d2b186e5e7dec3f4e5d6254b","Type":"ContainerStarted","Data":"48c57b8a2ecc0011563d0718760cd11c0c2a94d6e45f4ad9aad67c60b41c25f2"} Jul 14 11:37:49.479363 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.479317 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal" event={"ID":"ef29f282130d13891da160528c132f5f","Type":"ContainerStarted","Data":"9b4212ef5e457013b9c37fd099ed049008f63a46d730a72fe68a9389e4694320"} Jul 14 11:37:49.486672 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.486648 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 11:37:49.488877 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.488853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ff938fe-df60-4fb5-b558-e7832e86ff4b-serviceca\") pod \"node-ca-dgl7z\" (UID: \"1ff938fe-df60-4fb5-b558-e7832e86ff4b\") " pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.488993 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.488890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w79p2\" (UniqueName: \"kubernetes.io/projected/1ff938fe-df60-4fb5-b558-e7832e86ff4b-kube-api-access-w79p2\") pod \"node-ca-dgl7z\" (UID: \"1ff938fe-df60-4fb5-b558-e7832e86ff4b\") " pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.488993 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.488922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c881463-b72f-4fe7-a835-744f68a507f9-iptables-alerter-script\") pod \"iptables-alerter-gbmbj\" (UID: \"3c881463-b72f-4fe7-a835-744f68a507f9\") " pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.488993 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.488960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-run-netns\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.488993 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.488986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-conf-dir\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-slash\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7eac93e-56a6-4d90-aa2e-507659c30532-ovnkube-config\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-socket-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-run-netns\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-cni-dir\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc95de41-f270-4b20-b65a-57b877df87d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-kubelet-config\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-cni-bin\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed3a9e7-5412-4606-b01a-a852629061a5-tmp-dir\") pod \"node-resolver-tfd9j\" (UID: \"eed3a9e7-5412-4606-b01a-a852629061a5\") " pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.489228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-systemd\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-run-k8s-cni-cncf-io\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-var-lib-kubelet\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489319 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-daemon-config\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7eac93e-56a6-4d90-aa2e-507659c30532-env-overrides\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-device-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18eb4773-aa04-4688-9217-2d6bf662d310-etc-tuned\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18eb4773-aa04-4688-9217-2d6bf662d310-tmp\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-cnibin\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489492 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ff938fe-df60-4fb5-b558-e7832e86ff4b-serviceca\") pod \"node-ca-dgl7z\" (UID: \"1ff938fe-df60-4fb5-b558-e7832e86ff4b\") " pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-hostroot\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-hostroot\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489599 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-conf-dir\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489609 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c881463-b72f-4fe7-a835-744f68a507f9-iptables-alerter-script\") pod \"iptables-alerter-gbmbj\" (UID: \"3c881463-b72f-4fe7-a835-744f68a507f9\") " pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.489720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489615 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-var-lib-kubelet\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489833 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-device-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489859 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489898 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eed3a9e7-5412-4606-b01a-a852629061a5-hosts-file\") pod \"node-resolver-tfd9j\" (UID: \"eed3a9e7-5412-4606-b01a-a852629061a5\") " pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489944 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-kubelet-config\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489981 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-cni-bin\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490142 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-daemon-config\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-systemd\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-cnibin\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.489545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eed3a9e7-5412-4606-b01a-a852629061a5-hosts-file\") pod \"node-resolver-tfd9j\" (UID: \"eed3a9e7-5412-4606-b01a-a852629061a5\") " pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490320 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-kubernetes\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490330 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed3a9e7-5412-4606-b01a-a852629061a5-tmp-dir\") pod \"node-resolver-tfd9j\" (UID: \"eed3a9e7-5412-4606-b01a-a852629061a5\") " pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.490436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490436 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-kubernetes\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.491060 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490818 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-socket-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.491060 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc95de41-f270-4b20-b65a-57b877df87d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.491060 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490833 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-slash\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.491060 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490884 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-run-k8s-cni-cncf-io\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.491835 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.491808 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7eac93e-56a6-4d90-aa2e-507659c30532-env-overrides\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.492108 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492082 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-cni-dir\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.492312 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7eac93e-56a6-4d90-aa2e-507659c30532-ovnkube-config\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.492499 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.490340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:49.492606 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-socket-dir-parent\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.492606 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492569 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjg2\" (UniqueName: \"kubernetes.io/projected/fc95de41-f270-4b20-b65a-57b877df87d7-kube-api-access-mhjg2\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.492716 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492604 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-kubelet\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.492716 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c881463-b72f-4fe7-a835-744f68a507f9-host-slash\") pod \"iptables-alerter-gbmbj\" (UID: \"3c881463-b72f-4fe7-a835-744f68a507f9\") " pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.492716 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492665 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-host\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.492716 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-system-cni-dir\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.492908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492721 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-run-ovn\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.492908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492753 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rst\" (UniqueName: \"kubernetes.io/projected/3c881463-b72f-4fe7-a835-744f68a507f9-kube-api-access-p6rst\") pod \"iptables-alerter-gbmbj\" (UID: \"3c881463-b72f-4fe7-a835-744f68a507f9\") " pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.492908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492783 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-run\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.492908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-sys\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.492908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492837 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p27mf\" (UniqueName: \"kubernetes.io/projected/18eb4773-aa04-4688-9217-2d6bf662d310-kube-api-access-p27mf\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.492908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492866 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-run-openvswitch\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.492908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492898 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jhd\" (UniqueName: \"kubernetes.io/projected/f7eac93e-56a6-4d90-aa2e-507659c30532-kube-api-access-48jhd\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.493232 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ff938fe-df60-4fb5-b558-e7832e86ff4b-host\") pod \"node-ca-dgl7z\" (UID: \"1ff938fe-df60-4fb5-b558-e7832e86ff4b\") " pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.493232 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492955 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc95de41-f270-4b20-b65a-57b877df87d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.493232 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.492985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-registration-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.493232 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-sys-fs\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.493232 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vjw\" (UniqueName: \"kubernetes.io/projected/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-kube-api-access-h5vjw\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.493232 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493059 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-run-ovn\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.493232 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-sysctl-conf\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.493232 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493106 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-lib-modules\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.493232 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493148 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-var-lib-cni-bin\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493295 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-run-multus-certs\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8832e47-e0b9-43bc-ba93-3da7fef69778-cni-binary-copy\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493337 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-host\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493409 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-system-cni-dir\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-cnibin\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493356 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-cnibin\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493527 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/55ed8677-a1e3-4247-8b7f-0cca2d6cd535-agent-certs\") pod \"konnectivity-agent-tmggf\" (UID: \"55ed8677-a1e3-4247-8b7f-0cca2d6cd535\") " pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-etc-openvswitch\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493625 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7eac93e-56a6-4d90-aa2e-507659c30532-ovn-node-metrics-cert\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7eac93e-56a6-4d90-aa2e-507659c30532-ovnkube-script-lib\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.493690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqjq\" (UniqueName: \"kubernetes.io/projected/eed3a9e7-5412-4606-b01a-a852629061a5-kube-api-access-mdqjq\") pod \"node-resolver-tfd9j\" (UID: \"eed3a9e7-5412-4606-b01a-a852629061a5\") " pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493712 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-run\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6gg5\" (UniqueName: \"kubernetes.io/projected/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-kube-api-access-x6gg5\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493777 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-sys\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-etc-kubernetes\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc95de41-f270-4b20-b65a-57b877df87d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-run-netns\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-modprobe-d\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-os-release\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494050 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-dbus\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-systemd-units\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-sysconfig\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494169 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-sysctl-d\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-var-lib-kubelet\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.494253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494235 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-node-log\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-var-lib-cni-multus\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494270 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18eb4773-aa04-4688-9217-2d6bf662d310-tmp\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-run-ovn-kubernetes\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-cni-netd\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494345 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-run-multus-certs\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494383 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/55ed8677-a1e3-4247-8b7f-0cca2d6cd535-konnectivity-ca\") pod \"konnectivity-agent-tmggf\" (UID: \"55ed8677-a1e3-4247-8b7f-0cca2d6cd535\") " pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494410 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-cni-netd\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494414 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-run-systemd\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-log-socket\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.493354 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c881463-b72f-4fe7-a835-744f68a507f9-host-slash\") pod \"iptables-alerter-gbmbj\" (UID: \"3c881463-b72f-4fe7-a835-744f68a507f9\") " pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.494865 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.494743 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-multus-socket-dir-parent\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.495388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-run-systemd\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-sys-fs\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-run-openvswitch\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8832e47-e0b9-43bc-ba93-3da7fef69778-cni-binary-copy\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495547 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-os-release\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495583 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-dbus\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ff938fe-df60-4fb5-b558-e7832e86ff4b-host\") pod \"node-ca-dgl7z\" (UID: \"1ff938fe-df60-4fb5-b558-e7832e86ff4b\") " pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-sysconfig\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495674 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-systemd-units\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495683 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/55ed8677-a1e3-4247-8b7f-0cca2d6cd535-konnectivity-ca\") pod \"konnectivity-agent-tmggf\" (UID: \"55ed8677-a1e3-4247-8b7f-0cca2d6cd535\") " pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495733 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-kubelet\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495832 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-var-lib-cni-bin\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495939 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-run-ovn-kubernetes\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.495984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-host-var-lib-cni-multus\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.496677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496033 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-log-socket\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496105 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-sysctl-conf\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496112 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496204 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-sysctl-d\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8m8z\" (UniqueName: \"kubernetes.io/projected/b8832e47-e0b9-43bc-ba93-3da7fef69778-kube-api-access-m8m8z\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496259 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7eac93e-56a6-4d90-aa2e-507659c30532-ovnkube-script-lib\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-lib-modules\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496418 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-var-lib-kubelet\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.496510 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496529 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18eb4773-aa04-4688-9217-2d6bf662d310-etc-modprobe-d\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-var-lib-openvswitch\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.497528 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.496592 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret podName:03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc nodeName:}" failed. No retries permitted until 2026-07-14 11:37:49.99656357 +0000 UTC m=+3.072418157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret") pod "global-pull-secret-syncer-xkhkk" (UID: "03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc") : object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-os-release\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-var-lib-openvswitch\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-system-cni-dir\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496718 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-registration-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496763 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-node-log\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.496935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc95de41-f270-4b20-b65a-57b877df87d7-system-cni-dir\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.497019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-os-release\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.497110 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.497176 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc95de41-f270-4b20-b65a-57b877df87d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.497186 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs podName:eeb5ccae-f66d-42fe-bfca-8016ee9904f8 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:49.997142378 +0000 UTC m=+3.072996960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs") pod "network-metrics-daemon-t2c79" (UID: "eeb5ccae-f66d-42fe-bfca-8016ee9904f8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.497222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.497246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-etc-openvswitch\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.498260 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.497288 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7eac93e-56a6-4d90-aa2e-507659c30532-host-run-netns\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.498928 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.497556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8832e47-e0b9-43bc-ba93-3da7fef69778-etc-kubernetes\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.498928 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.498097 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc95de41-f270-4b20-b65a-57b877df87d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.498928 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.498776 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18eb4773-aa04-4688-9217-2d6bf662d310-etc-tuned\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.500284 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.500260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7eac93e-56a6-4d90-aa2e-507659c30532-ovn-node-metrics-cert\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.500518 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.500454 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/55ed8677-a1e3-4247-8b7f-0cca2d6cd535-agent-certs\") pod \"konnectivity-agent-tmggf\" (UID: \"55ed8677-a1e3-4247-8b7f-0cca2d6cd535\") " pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:37:49.500850 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.500828 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 11:37:49.500936 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.500856 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 11:37:49.500936 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.500872 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rwbpt for pod openshift-network-diagnostics/network-check-target-js79l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:49.500936 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:49.500929 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt podName:1c308823-050d-41d1-8aac-6cd004f01c3c nodeName:}" failed. No retries permitted until 2026-07-14 11:37:50.000912717 +0000 UTC m=+3.076767314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rwbpt" (UniqueName: "kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt") pod "network-check-target-js79l" (UID: "1c308823-050d-41d1-8aac-6cd004f01c3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:49.503571 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.503514 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w79p2\" (UniqueName: \"kubernetes.io/projected/1ff938fe-df60-4fb5-b558-e7832e86ff4b-kube-api-access-w79p2\") pod \"node-ca-dgl7z\" (UID: \"1ff938fe-df60-4fb5-b558-e7832e86ff4b\") " pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:49.507895 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.507844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rst\" (UniqueName: \"kubernetes.io/projected/3c881463-b72f-4fe7-a835-744f68a507f9-kube-api-access-p6rst\") pod \"iptables-alerter-gbmbj\" (UID: \"3c881463-b72f-4fe7-a835-744f68a507f9\") " pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.510363 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.510307 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p27mf\" (UniqueName: \"kubernetes.io/projected/18eb4773-aa04-4688-9217-2d6bf662d310-kube-api-access-p27mf\") pod \"tuned-c562x\" (UID: \"18eb4773-aa04-4688-9217-2d6bf662d310\") " pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.510839 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.510814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jhd\" (UniqueName: \"kubernetes.io/projected/f7eac93e-56a6-4d90-aa2e-507659c30532-kube-api-access-48jhd\") pod \"ovnkube-node-md69v\" (UID: \"f7eac93e-56a6-4d90-aa2e-507659c30532\") " pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.511381 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.511357 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6gg5\" (UniqueName: \"kubernetes.io/projected/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-kube-api-access-x6gg5\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:49.512236 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.512212 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqjq\" (UniqueName: \"kubernetes.io/projected/eed3a9e7-5412-4606-b01a-a852629061a5-kube-api-access-mdqjq\") pod \"node-resolver-tfd9j\" (UID: \"eed3a9e7-5412-4606-b01a-a852629061a5\") " pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.512860 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.512834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjg2\" (UniqueName: \"kubernetes.io/projected/fc95de41-f270-4b20-b65a-57b877df87d7-kube-api-access-mhjg2\") pod \"multus-additional-cni-plugins-zx9tj\" (UID: \"fc95de41-f270-4b20-b65a-57b877df87d7\") " pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.513178 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.513142 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vjw\" (UniqueName: \"kubernetes.io/projected/d11e012e-6be6-44b3-b5ae-eb03447fc5f2-kube-api-access-h5vjw\") pod \"aws-ebs-csi-driver-node-cgrkt\" (UID: \"d11e012e-6be6-44b3-b5ae-eb03447fc5f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.513512 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.513496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8m8z\" (UniqueName: \"kubernetes.io/projected/b8832e47-e0b9-43bc-ba93-3da7fef69778-kube-api-access-m8m8z\") pod \"multus-fjzx5\" (UID: \"b8832e47-e0b9-43bc-ba93-3da7fef69778\") " pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.591195 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.591150 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjzx5" Jul 14 11:37:49.599599 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.599573 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tfd9j" Jul 14 11:37:49.612348 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.612316 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbmbj" Jul 14 11:37:49.621341 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.621322 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" Jul 14 11:37:49.629263 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.629238 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c562x" Jul 14 11:37:49.638775 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.638755 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" Jul 14 11:37:49.645684 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.645665 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:37:49.652543 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.652525 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:37:49.658419 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:49.658398 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dgl7z" Jul 14 11:37:50.000276 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:50.000173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:50.000276 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:50.000222 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:50.000508 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:50.000348 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:50.000508 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:50.000382 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:50.000508 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:50.000436 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs podName:eeb5ccae-f66d-42fe-bfca-8016ee9904f8 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:51.000413539 +0000 UTC m=+4.076268151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs") pod "network-metrics-daemon-t2c79" (UID: "eeb5ccae-f66d-42fe-bfca-8016ee9904f8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:50.000508 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:50.000456 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret podName:03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc nodeName:}" failed. No retries permitted until 2026-07-14 11:37:51.000446164 +0000 UTC m=+4.076300761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret") pod "global-pull-secret-syncer-xkhkk" (UID: "03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc") : object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:50.101091 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:50.101058 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:50.101295 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:50.101267 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 11:37:50.101382 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:50.101302 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 11:37:50.101382 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:50.101318 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rwbpt for pod openshift-network-diagnostics/network-check-target-js79l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:50.101486 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:50.101387 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt podName:1c308823-050d-41d1-8aac-6cd004f01c3c nodeName:}" failed. No retries permitted until 2026-07-14 11:37:51.101365984 +0000 UTC m=+4.177220567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rwbpt" (UniqueName: "kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt") pod "network-check-target-js79l" (UID: "1c308823-050d-41d1-8aac-6cd004f01c3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:50.329991 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:50.329897 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 11:32:49 +0000 UTC" deadline="2028-02-11 09:45:10.94238272 +0000 UTC" Jul 14 11:37:50.329991 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:50.329944 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13846h7m20.612444212s" Jul 14 11:37:50.474326 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:50.474292 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:50.474425 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:50.474396 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:37:50.488337 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:50.488296 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c881463_b72f_4fe7_a835_744f68a507f9.slice/crio-1640bed55bc8fb766a0f2604c29d760e46a158c4640b49e3835ad484b322ff55 WatchSource:0}: Error finding container 1640bed55bc8fb766a0f2604c29d760e46a158c4640b49e3835ad484b322ff55: Status 404 returned error can't find the container with id 1640bed55bc8fb766a0f2604c29d760e46a158c4640b49e3835ad484b322ff55 Jul 14 11:37:50.489720 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:50.489678 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc95de41_f270_4b20_b65a_57b877df87d7.slice/crio-f6bb7468c4296976303b3bf5d5008594c1898f6f9575aa88e4d7f5124504601c WatchSource:0}: Error finding container f6bb7468c4296976303b3bf5d5008594c1898f6f9575aa88e4d7f5124504601c: Status 404 returned error can't find the container with id f6bb7468c4296976303b3bf5d5008594c1898f6f9575aa88e4d7f5124504601c Jul 14 11:37:50.490517 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:50.490489 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11e012e_6be6_44b3_b5ae_eb03447fc5f2.slice/crio-5cefd144d60c96ca69f793bf4580750c016bd12a534e62ea1e38550506156989 WatchSource:0}: Error finding container 5cefd144d60c96ca69f793bf4580750c016bd12a534e62ea1e38550506156989: Status 404 returned error can't find the container with id 5cefd144d60c96ca69f793bf4580750c016bd12a534e62ea1e38550506156989 Jul 14 11:37:50.494205 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:50.494183 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8832e47_e0b9_43bc_ba93_3da7fef69778.slice/crio-a685826c72b0654348504f4e34312500129d107a0ab89bbc55ee77cd82dd575b WatchSource:0}: Error finding container a685826c72b0654348504f4e34312500129d107a0ab89bbc55ee77cd82dd575b: Status 404 returned error can't find the container with id a685826c72b0654348504f4e34312500129d107a0ab89bbc55ee77cd82dd575b Jul 14 11:37:50.496735 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:37:50.495906 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7eac93e_56a6_4d90_aa2e_507659c30532.slice/crio-f13b3c261aa977927e4e86df2a59a0a7e077c1fc4e356d3e98ab90db5a1ea1d7 WatchSource:0}: Error finding container f13b3c261aa977927e4e86df2a59a0a7e077c1fc4e356d3e98ab90db5a1ea1d7: Status 404 returned error can't find the container with id f13b3c261aa977927e4e86df2a59a0a7e077c1fc4e356d3e98ab90db5a1ea1d7 Jul 14 11:37:51.008531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.007479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:51.008531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.007762 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:51.008531 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.007898 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:51.008531 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.007957 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret podName:03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc nodeName:}" failed. No retries permitted until 2026-07-14 11:37:53.00793856 +0000 UTC m=+6.083793143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret") pod "global-pull-secret-syncer-xkhkk" (UID: "03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc") : object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:51.008531 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.008363 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:51.008531 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.008415 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs podName:eeb5ccae-f66d-42fe-bfca-8016ee9904f8 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:53.008399323 +0000 UTC m=+6.084253895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs") pod "network-metrics-daemon-t2c79" (UID: "eeb5ccae-f66d-42fe-bfca-8016ee9904f8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:51.109127 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.109075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:51.109352 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.109331 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 11:37:51.109451 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.109360 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 11:37:51.109451 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.109374 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rwbpt for pod openshift-network-diagnostics/network-check-target-js79l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:51.109560 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.109457 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt podName:1c308823-050d-41d1-8aac-6cd004f01c3c nodeName:}" failed. No retries permitted until 2026-07-14 11:37:53.109416651 +0000 UTC m=+6.185271222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rwbpt" (UniqueName: "kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt") pod "network-check-target-js79l" (UID: "1c308823-050d-41d1-8aac-6cd004f01c3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:51.330886 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.330741 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 11:32:49 +0000 UTC" deadline="2028-03-20 13:04:39.175033143 +0000 UTC" Jul 14 11:37:51.330886 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.330804 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14761h26m47.844233849s" Jul 14 11:37:51.473424 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.473389 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:51.473598 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.473532 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:37:51.473996 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.473971 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:51.474098 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:51.474073 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:37:51.494605 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.494566 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dgl7z" event={"ID":"1ff938fe-df60-4fb5-b558-e7832e86ff4b","Type":"ContainerStarted","Data":"4a8e93dfd2c38f2ec4644d53d562bb022140c8fb1032048bbf266ed020c56583"} Jul 14 11:37:51.498107 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.498072 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" event={"ID":"d11e012e-6be6-44b3-b5ae-eb03447fc5f2","Type":"ContainerStarted","Data":"5cefd144d60c96ca69f793bf4580750c016bd12a534e62ea1e38550506156989"} Jul 14 11:37:51.504578 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.504524 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" event={"ID":"fc95de41-f270-4b20-b65a-57b877df87d7","Type":"ContainerStarted","Data":"f6bb7468c4296976303b3bf5d5008594c1898f6f9575aa88e4d7f5124504601c"} Jul 14 11:37:51.509140 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.509113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerStarted","Data":"f13b3c261aa977927e4e86df2a59a0a7e077c1fc4e356d3e98ab90db5a1ea1d7"} Jul 14 11:37:51.511920 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.511892 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjzx5" event={"ID":"b8832e47-e0b9-43bc-ba93-3da7fef69778","Type":"ContainerStarted","Data":"a685826c72b0654348504f4e34312500129d107a0ab89bbc55ee77cd82dd575b"} Jul 14 11:37:51.517384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.517355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbmbj" event={"ID":"3c881463-b72f-4fe7-a835-744f68a507f9","Type":"ContainerStarted","Data":"1640bed55bc8fb766a0f2604c29d760e46a158c4640b49e3835ad484b322ff55"} Jul 14 11:37:51.528013 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.527975 2562 generic.go:356] "Generic (PLEG): container finished" podID="20e26076d2b186e5e7dec3f4e5d6254b" containerID="8fdea03ecf26f02d8991ebf58b79adb34d00520bff2f9d73ed9da17ced5374e6" exitCode=0 Jul 14 11:37:51.528190 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.528106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" event={"ID":"20e26076d2b186e5e7dec3f4e5d6254b","Type":"ContainerDied","Data":"8fdea03ecf26f02d8991ebf58b79adb34d00520bff2f9d73ed9da17ced5374e6"} Jul 14 11:37:51.544531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.544041 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal" event={"ID":"ef29f282130d13891da160528c132f5f","Type":"ContainerStarted","Data":"c239a4925750042b354449c025c17d7f55a7b130b10c5d5837516ffa85a6f4e3"} Jul 14 11:37:51.553442 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.553159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c562x" event={"ID":"18eb4773-aa04-4688-9217-2d6bf662d310","Type":"ContainerStarted","Data":"1e5d286f58cba311673da6625a98f839a88e21470fe751f6c02d3f4344ddefd8"} Jul 14 11:37:51.567539 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.566934 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-116.ec2.internal" podStartSLOduration=2.566914498 podStartE2EDuration="2.566914498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:51.56659095 +0000 UTC m=+4.642445543" watchObservedRunningTime="2026-07-14 11:37:51.566914498 +0000 UTC m=+4.642769091" Jul 14 11:37:51.576284 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.576199 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tmggf" event={"ID":"55ed8677-a1e3-4247-8b7f-0cca2d6cd535","Type":"ContainerStarted","Data":"bf97d6440b0be3c2a7ebef36b02252e9401e21c8cc40705255bc93487c515171"} Jul 14 11:37:51.586088 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:51.586021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tfd9j" event={"ID":"eed3a9e7-5412-4606-b01a-a852629061a5","Type":"ContainerStarted","Data":"21716c7a53342c1299f3b112c059ad32717a63e22bcc3942ddd687d9cdcca12b"} Jul 14 11:37:52.474140 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:52.473650 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:52.474140 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:52.473880 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:37:52.608799 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:52.608758 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" event={"ID":"20e26076d2b186e5e7dec3f4e5d6254b","Type":"ContainerStarted","Data":"f6a8b1727a9ff13b28e87ac71df38f3d65d9569ab8851c3a20adb429199aa009"} Jul 14 11:37:52.629123 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:52.629061 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-116.ec2.internal" podStartSLOduration=3.629038612 podStartE2EDuration="3.629038612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:37:52.628215079 +0000 UTC m=+5.704069673" watchObservedRunningTime="2026-07-14 11:37:52.629038612 +0000 UTC m=+5.704893205" Jul 14 11:37:53.026291 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:53.026247 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:53.026504 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:53.026306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:53.026504 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.026448 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:53.026617 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.026601 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs podName:eeb5ccae-f66d-42fe-bfca-8016ee9904f8 nodeName:}" failed. No retries permitted until 2026-07-14 11:37:57.026579672 +0000 UTC m=+10.102434255 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs") pod "network-metrics-daemon-t2c79" (UID: "eeb5ccae-f66d-42fe-bfca-8016ee9904f8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:53.026967 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.026475 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:53.027047 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.027009 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret podName:03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc nodeName:}" failed. No retries permitted until 2026-07-14 11:37:57.026991968 +0000 UTC m=+10.102846552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret") pod "global-pull-secret-syncer-xkhkk" (UID: "03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc") : object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:53.128266 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:53.127582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:53.128266 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.127751 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 11:37:53.128266 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.127786 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 11:37:53.128266 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.127800 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rwbpt for pod openshift-network-diagnostics/network-check-target-js79l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:53.128266 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.127864 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt podName:1c308823-050d-41d1-8aac-6cd004f01c3c nodeName:}" failed. No retries permitted until 2026-07-14 11:37:57.127845194 +0000 UTC m=+10.203699764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rwbpt" (UniqueName: "kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt") pod "network-check-target-js79l" (UID: "1c308823-050d-41d1-8aac-6cd004f01c3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:53.476697 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:53.475918 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:53.476697 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.476053 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:37:53.476697 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:53.476464 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:53.476697 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:53.476583 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:37:54.474012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:54.473977 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:54.474519 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:54.474148 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:37:55.474936 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:55.474899 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:55.475324 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:55.475030 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:37:55.475626 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:55.475469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:55.475626 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:55.475590 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:37:56.473577 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:56.473388 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:56.473577 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:56.473526 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:37:57.061148 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:57.060641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:57.061148 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:57.060688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:57.061148 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.060809 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:57.061148 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.060822 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:57.061148 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.060875 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret podName:03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc nodeName:}" failed. No retries permitted until 2026-07-14 11:38:05.060857852 +0000 UTC m=+18.136712422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret") pod "global-pull-secret-syncer-xkhkk" (UID: "03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc") : object "kube-system"/"original-pull-secret" not registered Jul 14 11:37:57.061148 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.060895 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs podName:eeb5ccae-f66d-42fe-bfca-8016ee9904f8 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:05.060885497 +0000 UTC m=+18.136740069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs") pod "network-metrics-daemon-t2c79" (UID: "eeb5ccae-f66d-42fe-bfca-8016ee9904f8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:37:57.161876 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:57.161670 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:57.161876 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.161877 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 11:37:57.162179 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.161897 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 11:37:57.162179 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.161908 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rwbpt for pod openshift-network-diagnostics/network-check-target-js79l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:57.162179 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.161963 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt podName:1c308823-050d-41d1-8aac-6cd004f01c3c nodeName:}" failed. No retries permitted until 2026-07-14 11:38:05.161947288 +0000 UTC m=+18.237801862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rwbpt" (UniqueName: "kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt") pod "network-check-target-js79l" (UID: "1c308823-050d-41d1-8aac-6cd004f01c3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:37:57.474937 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:57.474888 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:57.475141 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.474990 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:37:57.475505 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:57.475356 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:57.475505 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:57.475463 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:37:58.473630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:58.473602 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:37:58.474025 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:58.473736 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:37:59.473893 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:59.473809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:37:59.474386 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:59.473942 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:37:59.474386 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:37:59.474029 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:37:59.474386 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:37:59.474173 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:00.473374 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:00.473267 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:00.473552 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:00.473404 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:01.473435 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:01.473407 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:01.473951 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:01.473522 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:01.473951 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:01.473575 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:01.473951 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:01.473706 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:02.473430 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:02.473402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:02.473584 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:02.473512 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:03.473864 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:03.473830 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:03.474291 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:03.473960 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:03.474291 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:03.473972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:03.474291 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:03.474102 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:04.474280 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:04.474243 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:04.474774 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:04.474374 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:05.126851 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:05.126822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:05.127027 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:05.126862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:05.127027 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.126979 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 11:38:05.127151 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.127032 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret podName:03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc nodeName:}" failed. No retries permitted until 2026-07-14 11:38:21.127019483 +0000 UTC m=+34.202874052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret") pod "global-pull-secret-syncer-xkhkk" (UID: "03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc") : object "kube-system"/"original-pull-secret" not registered Jul 14 11:38:05.127151 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.126979 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:38:05.127151 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.127073 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs podName:eeb5ccae-f66d-42fe-bfca-8016ee9904f8 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:21.127064102 +0000 UTC m=+34.202918681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs") pod "network-metrics-daemon-t2c79" (UID: "eeb5ccae-f66d-42fe-bfca-8016ee9904f8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:38:05.227806 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:05.227775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:05.227955 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.227912 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 11:38:05.227955 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.227933 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 11:38:05.227955 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.227943 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rwbpt for pod openshift-network-diagnostics/network-check-target-js79l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:38:05.228132 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.227988 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt podName:1c308823-050d-41d1-8aac-6cd004f01c3c nodeName:}" failed. No retries permitted until 2026-07-14 11:38:21.22797503 +0000 UTC m=+34.303829599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rwbpt" (UniqueName: "kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt") pod "network-check-target-js79l" (UID: "1c308823-050d-41d1-8aac-6cd004f01c3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:38:05.474186 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:05.474154 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:05.474333 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.474271 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:05.474677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:05.474439 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:05.474677 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:05.474571 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:06.473936 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:06.473900 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:06.474132 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:06.474026 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:07.477538 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:07.475972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:07.477538 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:07.476130 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:07.477538 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:07.476570 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:07.477538 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:07.476705 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:08.474274 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:08.474244 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:08.474435 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:08.474350 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:09.473926 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.473693 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:09.474576 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.473708 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:09.474576 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:09.474059 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:09.474576 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:09.474161 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:09.639221 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.639191 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tfd9j" event={"ID":"eed3a9e7-5412-4606-b01a-a852629061a5","Type":"ContainerStarted","Data":"6f0dacd07a3e99181010acaa19ebca6d8fa5642d4d4ec23c319ddb01ebe8001a"} Jul 14 11:38:09.640525 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.640496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dgl7z" event={"ID":"1ff938fe-df60-4fb5-b558-e7832e86ff4b","Type":"ContainerStarted","Data":"241f29c8784e28c1dc05a6290b773bad9a20aa261f0855c2d7f5e7b29a18b4d4"} Jul 14 11:38:09.641860 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.641838 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" event={"ID":"d11e012e-6be6-44b3-b5ae-eb03447fc5f2","Type":"ContainerStarted","Data":"3a39600561267991419e0e43e997d6f79759e1ec9e5c7e4c0553e96f6a195c73"} Jul 14 11:38:09.643204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.643182 2562 generic.go:356] "Generic (PLEG): container finished" podID="fc95de41-f270-4b20-b65a-57b877df87d7" containerID="3c04b2fd3f4a42dea1968ab3b3b0414a4c03e56d36f5da51f75d6eadc87025a0" exitCode=0 Jul 14 11:38:09.643297 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.643207 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" event={"ID":"fc95de41-f270-4b20-b65a-57b877df87d7","Type":"ContainerDied","Data":"3c04b2fd3f4a42dea1968ab3b3b0414a4c03e56d36f5da51f75d6eadc87025a0"} Jul 14 11:38:09.645195 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.645177 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:38:09.645530 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.645508 2562 generic.go:356] "Generic (PLEG): container finished" podID="f7eac93e-56a6-4d90-aa2e-507659c30532" containerID="049fe68445c1379585984266f73f6cc6938dec861198dbe623f1a68a1371bbe5" exitCode=1 Jul 14 11:38:09.645585 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.645562 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerStarted","Data":"bd009dbe734d03a18375361a5ae9fa5591f75a34d3ce024f307f9ae78cf079ba"} Jul 14 11:38:09.645640 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.645625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerDied","Data":"049fe68445c1379585984266f73f6cc6938dec861198dbe623f1a68a1371bbe5"} Jul 14 11:38:09.645693 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.645680 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerStarted","Data":"5b54adf6a93bf9c4ec1ca27c041badc711d7de3ce7e7f934cf89ef027ba86003"} Jul 14 11:38:09.647189 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.647171 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjzx5" event={"ID":"b8832e47-e0b9-43bc-ba93-3da7fef69778","Type":"ContainerStarted","Data":"54ff6153825fa3c13547d4aae5b86a88bed69a8038a6f533506f77bae77d68b2"} Jul 14 11:38:09.648606 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.648585 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c562x" event={"ID":"18eb4773-aa04-4688-9217-2d6bf662d310","Type":"ContainerStarted","Data":"023e82dc6f65d0c4bedc83b38545f24b454eb8604a9a56d946adc3dd7a4dfb2a"} Jul 14 11:38:09.650517 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.650500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tmggf" event={"ID":"55ed8677-a1e3-4247-8b7f-0cca2d6cd535","Type":"ContainerStarted","Data":"4b7f0b6043efbafe4d72961e004388cd403ecf3e33ec00046bd1f28789f6cb52"} Jul 14 11:38:09.685307 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.685274 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tfd9j" podStartSLOduration=3.005136767 podStartE2EDuration="21.685260702s" totalImagesPullingTime="18.680123935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:09.659144055 +0000 UTC m=+22.734998644" watchObservedRunningTime="2026-07-14 11:38:09.685260702 +0000 UTC m=+22.761115289" Jul 14 11:38:09.704227 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.704190 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tmggf" podStartSLOduration=11.143450139 podStartE2EDuration="21.70417812s" totalImagesPullingTime="10.560727981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:09.703969043 +0000 UTC m=+22.779823631" watchObservedRunningTime="2026-07-14 11:38:09.70417812 +0000 UTC m=+22.780032707" Jul 14 11:38:09.723573 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.723528 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dgl7z" podStartSLOduration=3.044878751 podStartE2EDuration="21.723508119s" totalImagesPullingTime="18.678629368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:09.72251868 +0000 UTC m=+22.798373267" watchObservedRunningTime="2026-07-14 11:38:09.723508119 +0000 UTC m=+22.799362708" Jul 14 11:38:09.747487 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.747451 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fjzx5" podStartSLOduration=3.010157146 podStartE2EDuration="21.747440513s" totalImagesPullingTime="18.737283367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:09.747409414 +0000 UTC m=+22.823264013" watchObservedRunningTime="2026-07-14 11:38:09.747440513 +0000 UTC m=+22.823295099" Jul 14 11:38:09.773119 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:09.773070 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c562x" podStartSLOduration=3.09342898 podStartE2EDuration="21.773058029s" totalImagesPullingTime="18.679629049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:09.772542116 +0000 UTC m=+22.848396704" watchObservedRunningTime="2026-07-14 11:38:09.773058029 +0000 UTC m=+22.848912621" Jul 14 11:38:10.474137 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:10.474110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:10.474541 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:10.474204 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:10.651275 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:10.651236 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 11:38:10.655550 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:10.655518 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" event={"ID":"d11e012e-6be6-44b3-b5ae-eb03447fc5f2","Type":"ContainerStarted","Data":"57b4327e01ad4e2a71008e830200b59ef7816d0df71815a3f82adbd4004248f6"} Jul 14 11:38:10.658830 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:10.658802 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:38:10.659314 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:10.659288 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerStarted","Data":"18cd03328ecb3d52ea2e3c2ee9627de9f1fb5d59a09d2de6f22ddae3b05c3e0b"} Jul 14 11:38:10.659408 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:10.659380 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerStarted","Data":"2af723aea0314996da1bd0cecf84dc3e877baa2d88fe296fa766f9d32a9fbe11"} Jul 14 11:38:10.659474 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:10.659461 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerStarted","Data":"15364d48da3f4ce9ffbbd0ff1af49a1f457b2663cb9b9a348943c892827c5d1a"} Jul 14 11:38:10.661004 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:10.660977 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbmbj" event={"ID":"3c881463-b72f-4fe7-a835-744f68a507f9","Type":"ContainerStarted","Data":"9c16e3176e20d360b5cba747de0dac6f6f32e9a6be68e6ca81d2f2bfdc4259cf"} Jul 14 11:38:10.682621 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:10.682582 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gbmbj" podStartSLOduration=3.992577236 podStartE2EDuration="22.682569888s" totalImagesPullingTime="18.689992652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:10.682556877 +0000 UTC m=+23.758411465" watchObservedRunningTime="2026-07-14 11:38:10.682569888 +0000 UTC m=+23.758424475" Jul 14 11:38:11.363180 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:11.363041 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T11:38:10.651255208Z","UUID":"15234fe0-5d9f-48db-9f62-ff61891bbf35","Handler":null,"Name":"","Endpoint":""} Jul 14 11:38:11.364701 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:11.364662 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 11:38:11.364844 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:11.364715 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 11:38:11.473953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:11.473927 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:11.473953 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:11.473962 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:11.474187 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:11.474050 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:11.474187 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:11.474166 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:12.473568 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:12.473391 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:12.473851 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:12.473647 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:12.667603 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:12.667571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" event={"ID":"d11e012e-6be6-44b3-b5ae-eb03447fc5f2","Type":"ContainerStarted","Data":"692b4c7b7c940db944bcc995c4596646c6ae496fbb642d6fbf7a0307ea1cca42"} Jul 14 11:38:12.670877 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:12.670855 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:38:12.671273 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:12.671246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerStarted","Data":"3f50f464dab9be8c67002145841dc2e50034f343d451d8d46f36049204365b45"} Jul 14 11:38:12.691967 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:12.691917 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgrkt" podStartSLOduration=3.731302451 podStartE2EDuration="24.691900245s" totalImagesPullingTime="20.960597794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:12.691315903 +0000 UTC m=+25.767170493" watchObservedRunningTime="2026-07-14 11:38:12.691900245 +0000 UTC m=+25.767754835" Jul 14 11:38:13.156702 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:13.156671 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:38:13.157336 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:13.157303 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:38:13.473498 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:13.473471 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:13.473671 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:13.473471 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:13.473671 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:13.473598 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:13.473782 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:13.473673 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:13.674452 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:13.674426 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:38:13.674945 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:13.674846 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tmggf" Jul 14 11:38:14.473609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.473447 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:14.473761 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:14.473686 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:14.677585 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.677512 2562 generic.go:356] "Generic (PLEG): container finished" podID="fc95de41-f270-4b20-b65a-57b877df87d7" containerID="7f7df6008eee300b9ed13c1e6b2d7950ae97cde2a00879ab56704898bc51d038" exitCode=0 Jul 14 11:38:14.678313 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.677596 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" event={"ID":"fc95de41-f270-4b20-b65a-57b877df87d7","Type":"ContainerDied","Data":"7f7df6008eee300b9ed13c1e6b2d7950ae97cde2a00879ab56704898bc51d038"} Jul 14 11:38:14.681052 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.681035 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:38:14.681417 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.681396 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerStarted","Data":"ce00e9918b17d971b7618edf7c14202ac81dd9ae17a89465392dd89589b8df05"} Jul 14 11:38:14.681680 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.681663 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:38:14.681793 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.681732 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:38:14.681859 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.681848 2562 scope.go:117] "RemoveContainer" containerID="049fe68445c1379585984266f73f6cc6938dec861198dbe623f1a68a1371bbe5" Jul 14 11:38:14.698228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.698203 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:38:14.698512 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:14.698497 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:38:15.473723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:15.473687 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:15.473903 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:15.473695 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:15.473903 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:15.473833 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:15.473993 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:15.473878 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:15.686873 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:15.686810 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:38:15.687266 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:15.687231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" event={"ID":"f7eac93e-56a6-4d90-aa2e-507659c30532","Type":"ContainerStarted","Data":"465e8ddd29c244bbc0de733f2177c3c75b81832d2381603ebcb4442ba8312bbe"} Jul 14 11:38:15.687306 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:15.687279 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 11:38:15.722325 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:15.722288 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" podStartSLOduration=8.966906870999999 podStartE2EDuration="27.722275589s" totalImagesPullingTime="18.755368718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:15.722111976 +0000 UTC m=+28.797966569" watchObservedRunningTime="2026-07-14 11:38:15.722275589 +0000 UTC m=+28.798130221" Jul 14 11:38:16.343716 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.343575 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:38:16.473270 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.473237 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:16.473416 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:16.473339 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:16.473947 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.473927 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-js79l"] Jul 14 11:38:16.474024 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.474014 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:16.474132 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:16.474113 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:16.485134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.485114 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xkhkk"] Jul 14 11:38:16.485614 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.485596 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t2c79"] Jul 14 11:38:16.485704 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.485692 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:16.485822 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:16.485789 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:16.691335 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.691258 2562 generic.go:356] "Generic (PLEG): container finished" podID="fc95de41-f270-4b20-b65a-57b877df87d7" containerID="2afb890b28705b84e7efd090a423b5b9a48036ce7c5905d46086385a0c24a824" exitCode=0 Jul 14 11:38:16.691335 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.691315 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:16.691335 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:16.691311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" event={"ID":"fc95de41-f270-4b20-b65a-57b877df87d7","Type":"ContainerDied","Data":"2afb890b28705b84e7efd090a423b5b9a48036ce7c5905d46086385a0c24a824"} Jul 14 11:38:16.691798 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:16.691398 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:18.473863 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:18.473820 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:18.474479 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:18.473843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:18.474479 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:18.473929 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:18.474479 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:18.473934 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:18.474479 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:18.474015 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:18.474479 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:18.474128 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:18.697925 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:18.697851 2562 generic.go:356] "Generic (PLEG): container finished" podID="fc95de41-f270-4b20-b65a-57b877df87d7" containerID="baf3bf4e9b6e6745a4255f91a93bafdba507e8f2c98ae303f89a51490ffe60d8" exitCode=0 Jul 14 11:38:18.697925 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:18.697917 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" event={"ID":"fc95de41-f270-4b20-b65a-57b877df87d7","Type":"ContainerDied","Data":"baf3bf4e9b6e6745a4255f91a93bafdba507e8f2c98ae303f89a51490ffe60d8"} Jul 14 11:38:20.473302 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:20.473238 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:20.473302 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:20.473282 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:20.473932 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:20.473400 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkhkk" podUID="03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc" Jul 14 11:38:20.473932 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:20.473435 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-js79l" podUID="1c308823-050d-41d1-8aac-6cd004f01c3c" Jul 14 11:38:20.473932 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:20.473461 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:20.473932 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:20.473554 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t2c79" podUID="eeb5ccae-f66d-42fe-bfca-8016ee9904f8" Jul 14 11:38:21.151091 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.150857 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:21.151312 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.151123 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:21.151312 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:21.151011 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:38:21.151312 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:21.151240 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs podName:eeb5ccae-f66d-42fe-bfca-8016ee9904f8 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:53.151218943 +0000 UTC m=+66.227073509 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs") pod "network-metrics-daemon-t2c79" (UID: "eeb5ccae-f66d-42fe-bfca-8016ee9904f8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 11:38:21.151312 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:21.151244 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 11:38:21.151312 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:21.151282 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret podName:03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc nodeName:}" failed. No retries permitted until 2026-07-14 11:38:53.151271283 +0000 UTC m=+66.227125849 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret") pod "global-pull-secret-syncer-xkhkk" (UID: "03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc") : object "kube-system"/"original-pull-secret" not registered Jul 14 11:38:21.251542 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.251515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:21.251675 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:21.251636 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 11:38:21.251675 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:21.251648 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 11:38:21.251675 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:21.251657 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rwbpt for pod openshift-network-diagnostics/network-check-target-js79l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:38:21.251774 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:21.251703 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt podName:1c308823-050d-41d1-8aac-6cd004f01c3c nodeName:}" failed. No retries permitted until 2026-07-14 11:38:53.251688278 +0000 UTC m=+66.327542844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rwbpt" (UniqueName: "kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt") pod "network-check-target-js79l" (UID: "1c308823-050d-41d1-8aac-6cd004f01c3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 11:38:21.732452 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.732423 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-116.ec2.internal" event="NodeReady" Jul 14 11:38:21.732861 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.732580 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 11:38:21.779454 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.779424 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-246h5"] Jul 14 11:38:21.809688 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.809621 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5bdxl"] Jul 14 11:38:21.809838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.809744 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-246h5" Jul 14 11:38:21.812791 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.812769 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9dtrp\"" Jul 14 11:38:21.812931 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.812766 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 11:38:21.812931 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.812769 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 11:38:21.823428 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.823408 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-246h5"] Jul 14 11:38:21.823541 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.823444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5bdxl"] Jul 14 11:38:21.823600 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.823532 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5bdxl" Jul 14 11:38:21.826447 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.826426 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 11:38:21.826609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.826567 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8ht88\"" Jul 14 11:38:21.826609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.826577 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 11:38:21.826720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.826624 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 11:38:21.955812 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.955723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96d809e2-9c08-4cb9-8a57-2c3aef340d37-cert\") pod \"ingress-canary-5bdxl\" (UID: \"96d809e2-9c08-4cb9-8a57-2c3aef340d37\") " pod="openshift-ingress-canary/ingress-canary-5bdxl" Jul 14 11:38:21.955812 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.955775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-config-volume\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:21.956047 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.955849 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnx74\" (UniqueName: \"kubernetes.io/projected/96d809e2-9c08-4cb9-8a57-2c3aef340d37-kube-api-access-gnx74\") pod \"ingress-canary-5bdxl\" (UID: \"96d809e2-9c08-4cb9-8a57-2c3aef340d37\") " pod="openshift-ingress-canary/ingress-canary-5bdxl" Jul 14 11:38:21.956047 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.955928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:21.956047 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.955949 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-tmp-dir\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:21.956047 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:21.956038 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rgw\" (UniqueName: \"kubernetes.io/projected/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-kube-api-access-w8rgw\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:22.056997 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.056968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rgw\" (UniqueName: \"kubernetes.io/projected/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-kube-api-access-w8rgw\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:22.056997 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.057003 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96d809e2-9c08-4cb9-8a57-2c3aef340d37-cert\") pod \"ingress-canary-5bdxl\" (UID: \"96d809e2-9c08-4cb9-8a57-2c3aef340d37\") " pod="openshift-ingress-canary/ingress-canary-5bdxl" Jul 14 11:38:22.057234 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.057031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-config-volume\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:22.057334 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.057306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnx74\" (UniqueName: \"kubernetes.io/projected/96d809e2-9c08-4cb9-8a57-2c3aef340d37-kube-api-access-gnx74\") pod \"ingress-canary-5bdxl\" (UID: \"96d809e2-9c08-4cb9-8a57-2c3aef340d37\") " pod="openshift-ingress-canary/ingress-canary-5bdxl" Jul 14 11:38:22.057467 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.057375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:22.057467 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.057400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-tmp-dir\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:22.057737 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:22.057656 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 11:38:22.057737 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.057675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-tmp-dir\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:22.057891 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:22.057747 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls podName:ec672a19-4fb9-410f-b2c5-7b6ba92830e4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:22.557726386 +0000 UTC m=+35.633580963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls") pod "dns-default-246h5" (UID: "ec672a19-4fb9-410f-b2c5-7b6ba92830e4") : secret "dns-default-metrics-tls" not found Jul 14 11:38:22.057891 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.057764 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-config-volume\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:22.061614 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.061595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96d809e2-9c08-4cb9-8a57-2c3aef340d37-cert\") pod \"ingress-canary-5bdxl\" (UID: \"96d809e2-9c08-4cb9-8a57-2c3aef340d37\") " pod="openshift-ingress-canary/ingress-canary-5bdxl" Jul 14 11:38:22.067007 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.066986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rgw\" (UniqueName: \"kubernetes.io/projected/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-kube-api-access-w8rgw\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:22.067238 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.067218 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnx74\" (UniqueName: \"kubernetes.io/projected/96d809e2-9c08-4cb9-8a57-2c3aef340d37-kube-api-access-gnx74\") pod \"ingress-canary-5bdxl\" (UID: \"96d809e2-9c08-4cb9-8a57-2c3aef340d37\") " pod="openshift-ingress-canary/ingress-canary-5bdxl" Jul 14 11:38:22.132143 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.132109 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5bdxl" Jul 14 11:38:22.473485 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.473447 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:22.473669 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.473497 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:22.473669 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.473458 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:22.478782 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.478728 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-26j7k\"" Jul 14 11:38:22.478782 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.478732 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 11:38:22.478977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.478817 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 11:38:22.478977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.478847 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 11:38:22.478977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.478850 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9256t\"" Jul 14 11:38:22.478977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.478874 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 11:38:22.561027 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.560987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:22.561202 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:22.561148 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 11:38:22.561256 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:22.561219 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls podName:ec672a19-4fb9-410f-b2c5-7b6ba92830e4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:23.561202733 +0000 UTC m=+36.637057299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls") pod "dns-default-246h5" (UID: "ec672a19-4fb9-410f-b2c5-7b6ba92830e4") : secret "dns-default-metrics-tls" not found Jul 14 11:38:22.987890 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:22.987861 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5bdxl"] Jul 14 11:38:23.047633 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:38:23.047600 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d809e2_9c08_4cb9_8a57_2c3aef340d37.slice/crio-62a1c5205a64c896401db050862f4e56a62aeed5661626b08d9415ca8eb9e6ad WatchSource:0}: Error finding container 62a1c5205a64c896401db050862f4e56a62aeed5661626b08d9415ca8eb9e6ad: Status 404 returned error can't find the container with id 62a1c5205a64c896401db050862f4e56a62aeed5661626b08d9415ca8eb9e6ad Jul 14 11:38:23.568854 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:23.568669 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:23.568854 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:23.568852 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 11:38:23.569048 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:23.568916 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls podName:ec672a19-4fb9-410f-b2c5-7b6ba92830e4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:25.568900776 +0000 UTC m=+38.644755345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls") pod "dns-default-246h5" (UID: "ec672a19-4fb9-410f-b2c5-7b6ba92830e4") : secret "dns-default-metrics-tls" not found Jul 14 11:38:23.711059 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:23.711020 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5bdxl" event={"ID":"96d809e2-9c08-4cb9-8a57-2c3aef340d37","Type":"ContainerStarted","Data":"62a1c5205a64c896401db050862f4e56a62aeed5661626b08d9415ca8eb9e6ad"} Jul 14 11:38:23.714132 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:23.714094 2562 generic.go:356] "Generic (PLEG): container finished" podID="fc95de41-f270-4b20-b65a-57b877df87d7" containerID="428e88eea6001a0a13463c814bc2ff86e6cab135c644efed26f4f7b370e25777" exitCode=0 Jul 14 11:38:23.714132 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:23.714110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" event={"ID":"fc95de41-f270-4b20-b65a-57b877df87d7","Type":"ContainerDied","Data":"428e88eea6001a0a13463c814bc2ff86e6cab135c644efed26f4f7b370e25777"} Jul 14 11:38:24.721286 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:24.721256 2562 generic.go:356] "Generic (PLEG): container finished" podID="fc95de41-f270-4b20-b65a-57b877df87d7" containerID="30f0b7a1a7e071d47b787e5f0e0daf3d8f491ee6c274c5ff81f63f3891167e38" exitCode=0 Jul 14 11:38:24.721721 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:24.721310 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" event={"ID":"fc95de41-f270-4b20-b65a-57b877df87d7","Type":"ContainerDied","Data":"30f0b7a1a7e071d47b787e5f0e0daf3d8f491ee6c274c5ff81f63f3891167e38"} Jul 14 11:38:25.586511 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:25.586422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:25.586638 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:25.586610 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 11:38:25.586707 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:25.586694 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls podName:ec672a19-4fb9-410f-b2c5-7b6ba92830e4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:29.586673202 +0000 UTC m=+42.662527788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls") pod "dns-default-246h5" (UID: "ec672a19-4fb9-410f-b2c5-7b6ba92830e4") : secret "dns-default-metrics-tls" not found Jul 14 11:38:25.726750 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:25.726717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" event={"ID":"fc95de41-f270-4b20-b65a-57b877df87d7","Type":"ContainerStarted","Data":"e4a3e375c49f881ea3afd66efebed07696c391578b18931411d161759dfc8135"} Jul 14 11:38:25.728108 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:25.728066 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5bdxl" event={"ID":"96d809e2-9c08-4cb9-8a57-2c3aef340d37","Type":"ContainerStarted","Data":"e734c15e859257e63dea17e75396463943b0010cca46c1f33b8c391133b904d2"} Jul 14 11:38:25.758708 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:25.758656 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zx9tj" podStartSLOduration=8.142254911 podStartE2EDuration="37.758638129s" totalImagesPullingTime="29.616383218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:25.756454131 +0000 UTC m=+38.832308719" watchObservedRunningTime="2026-07-14 11:38:25.758638129 +0000 UTC m=+38.834492749" Jul 14 11:38:25.784122 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:25.784056 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5bdxl" podStartSLOduration=2.470539113 podStartE2EDuration="4.784038216s" totalImagesPullingTime="2.313499103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:25.78348697 +0000 UTC m=+38.859341558" watchObservedRunningTime="2026-07-14 11:38:25.784038216 +0000 UTC m=+38.859892801" Jul 14 11:38:29.614612 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:29.614581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:29.615016 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:29.614683 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 11:38:29.615016 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:29.614728 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls podName:ec672a19-4fb9-410f-b2c5-7b6ba92830e4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:37.614716146 +0000 UTC m=+50.690570712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls") pod "dns-default-246h5" (UID: "ec672a19-4fb9-410f-b2c5-7b6ba92830e4") : secret "dns-default-metrics-tls" not found Jul 14 11:38:37.664536 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:37.664494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:37.665107 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:37.664613 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 11:38:37.665107 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:37.664693 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls podName:ec672a19-4fb9-410f-b2c5-7b6ba92830e4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:53.664673627 +0000 UTC m=+66.740528212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls") pod "dns-default-246h5" (UID: "ec672a19-4fb9-410f-b2c5-7b6ba92830e4") : secret "dns-default-metrics-tls" not found Jul 14 11:38:47.702652 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:47.702627 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-md69v" Jul 14 11:38:51.515775 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.515735 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q"] Jul 14 11:38:51.518767 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.518669 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q" Jul 14 11:38:51.521761 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.521730 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 14 11:38:51.521761 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.521758 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nj9lf\"" Jul 14 11:38:51.522204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.521805 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 14 11:38:51.535493 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.535469 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q"] Jul 14 11:38:51.556057 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.556032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsdw\" (UniqueName: \"kubernetes.io/projected/91564ec5-d937-4fb6-a637-04f8a18975bd-kube-api-access-vdsdw\") pod \"volume-data-source-validator-78495597f6-lnt4q\" (UID: \"91564ec5-d937-4fb6-a637-04f8a18975bd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q" Jul 14 11:38:51.621093 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.621053 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7894875846-lcf2d"] Jul 14 11:38:51.622838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.622823 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.624422 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.624401 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-k5nq8"] Jul 14 11:38:51.625901 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.625880 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.626854 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.626832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 14 11:38:51.627006 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.626988 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 11:38:51.627098 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.627048 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 14 11:38:51.627098 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.627063 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 11:38:51.627220 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.627131 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 14 11:38:51.627350 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.627333 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 14 11:38:51.627419 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.627404 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vj6tw\"" Jul 14 11:38:51.628778 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.628757 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 14 11:38:51.628911 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.628893 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 14 11:38:51.629357 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.629337 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ldh4n\"" Jul 14 11:38:51.629609 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.629449 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 14 11:38:51.629669 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.629618 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 14 11:38:51.634804 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.634783 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 14 11:38:51.642401 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.642378 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7894875846-lcf2d"] Jul 14 11:38:51.642608 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.642592 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-k5nq8"] Jul 14 11:38:51.656302 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsdw\" (UniqueName: \"kubernetes.io/projected/91564ec5-d937-4fb6-a637-04f8a18975bd-kube-api-access-vdsdw\") pod \"volume-data-source-validator-78495597f6-lnt4q\" (UID: \"91564ec5-d937-4fb6-a637-04f8a18975bd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q" Jul 14 11:38:51.656407 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-default-certificate\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.656407 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656340 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a500e755-d871-4992-9b39-18ef1f7e9330-trusted-ca\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.656407 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.656407 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc54l\" (UniqueName: \"kubernetes.io/projected/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-kube-api-access-xc54l\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.656601 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656492 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57n9t\" (UniqueName: \"kubernetes.io/projected/a500e755-d871-4992-9b39-18ef1f7e9330-kube-api-access-57n9t\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.656601 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656523 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a500e755-d871-4992-9b39-18ef1f7e9330-serving-cert\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.656601 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656580 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-stats-auth\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.656601 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656599 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.656726 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.656614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a500e755-d871-4992-9b39-18ef1f7e9330-config\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.667562 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.667536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsdw\" (UniqueName: \"kubernetes.io/projected/91564ec5-d937-4fb6-a637-04f8a18975bd-kube-api-access-vdsdw\") pod \"volume-data-source-validator-78495597f6-lnt4q\" (UID: \"91564ec5-d937-4fb6-a637-04f8a18975bd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q" Jul 14 11:38:51.757526 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.757503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a500e755-d871-4992-9b39-18ef1f7e9330-trusted-ca\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.757673 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.757529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.757673 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.757545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc54l\" (UniqueName: \"kubernetes.io/projected/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-kube-api-access-xc54l\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.757673 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.757577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57n9t\" (UniqueName: \"kubernetes.io/projected/a500e755-d871-4992-9b39-18ef1f7e9330-kube-api-access-57n9t\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.757673 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.757594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a500e755-d871-4992-9b39-18ef1f7e9330-serving-cert\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.757673 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.757637 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-stats-auth\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.757673 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.757654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.757673 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.757668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a500e755-d871-4992-9b39-18ef1f7e9330-config\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.757957 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:51.757686 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:52.257665999 +0000 UTC m=+65.333520602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : configmap references non-existent config key: service-ca.crt Jul 14 11:38:51.757957 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.757747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-default-certificate\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.757957 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:51.757789 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 11:38:51.757957 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:51.757852 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:52.257835476 +0000 UTC m=+65.333690043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : secret "router-metrics-certs-default" not found Jul 14 11:38:51.758339 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.758319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a500e755-d871-4992-9b39-18ef1f7e9330-config\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.758501 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.758482 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a500e755-d871-4992-9b39-18ef1f7e9330-trusted-ca\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.759912 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.759894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a500e755-d871-4992-9b39-18ef1f7e9330-serving-cert\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.759985 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.759950 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-stats-auth\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.760030 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.759992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-default-certificate\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.767302 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.767254 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc54l\" (UniqueName: \"kubernetes.io/projected/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-kube-api-access-xc54l\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:51.767402 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.767381 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57n9t\" (UniqueName: \"kubernetes.io/projected/a500e755-d871-4992-9b39-18ef1f7e9330-kube-api-access-57n9t\") pod \"console-operator-7866497494-k5nq8\" (UID: \"a500e755-d871-4992-9b39-18ef1f7e9330\") " pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.828785 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.828765 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q" Jul 14 11:38:51.938243 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.938216 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:38:51.945869 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:51.945845 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q"] Jul 14 11:38:51.948664 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:38:51.948642 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91564ec5_d937_4fb6_a637_04f8a18975bd.slice/crio-6da06be3b4ab06a51a62ca662f4af0988240f5443b97dcb52c696ef18aa3a0c4 WatchSource:0}: Error finding container 6da06be3b4ab06a51a62ca662f4af0988240f5443b97dcb52c696ef18aa3a0c4: Status 404 returned error can't find the container with id 6da06be3b4ab06a51a62ca662f4af0988240f5443b97dcb52c696ef18aa3a0c4 Jul 14 11:38:52.056559 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.056408 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-k5nq8"] Jul 14 11:38:52.058857 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:38:52.058833 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda500e755_d871_4992_9b39_18ef1f7e9330.slice/crio-ba739f378ea2ea8b3bf0174067f797892eb78e8d959a15f78751474459198390 WatchSource:0}: Error finding container ba739f378ea2ea8b3bf0174067f797892eb78e8d959a15f78751474459198390: Status 404 returned error can't find the container with id ba739f378ea2ea8b3bf0174067f797892eb78e8d959a15f78751474459198390 Jul 14 11:38:52.260835 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.260810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:52.260922 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.260850 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:52.260960 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:52.260947 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 11:38:52.260991 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:52.260980 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:53.260963393 +0000 UTC m=+66.336817979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : configmap references non-existent config key: service-ca.crt Jul 14 11:38:52.261029 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:52.260997 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:53.260988597 +0000 UTC m=+66.336843163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : secret "router-metrics-certs-default" not found Jul 14 11:38:52.394022 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.393970 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp"] Jul 14 11:38:52.396709 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.396694 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.399493 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.399473 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 14 11:38:52.399631 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.399554 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mrxlw\"" Jul 14 11:38:52.399725 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.399661 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 14 11:38:52.399725 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.399701 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 14 11:38:52.399901 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.399887 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 14 11:38:52.407055 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.407038 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp"] Jul 14 11:38:52.462665 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.462638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-wnpqp\" (UID: \"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.462770 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.462688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjcth\" (UniqueName: \"kubernetes.io/projected/d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0-kube-api-access-hjcth\") pod \"kube-storage-version-migrator-operator-799cb7f958-wnpqp\" (UID: \"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.462810 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.462769 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-wnpqp\" (UID: \"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.563588 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.563563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjcth\" (UniqueName: \"kubernetes.io/projected/d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0-kube-api-access-hjcth\") pod \"kube-storage-version-migrator-operator-799cb7f958-wnpqp\" (UID: \"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.563957 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.563641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-wnpqp\" (UID: \"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.563957 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.563707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-wnpqp\" (UID: \"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.564300 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.564276 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-wnpqp\" (UID: \"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.566214 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.566189 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-wnpqp\" (UID: \"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.573665 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.573644 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjcth\" (UniqueName: \"kubernetes.io/projected/d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0-kube-api-access-hjcth\") pod \"kube-storage-version-migrator-operator-799cb7f958-wnpqp\" (UID: \"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.706922 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.706863 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" Jul 14 11:38:52.788201 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.788166 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-k5nq8" event={"ID":"a500e755-d871-4992-9b39-18ef1f7e9330","Type":"ContainerStarted","Data":"ba739f378ea2ea8b3bf0174067f797892eb78e8d959a15f78751474459198390"} Jul 14 11:38:52.789700 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.789671 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q" event={"ID":"91564ec5-d937-4fb6-a637-04f8a18975bd","Type":"ContainerStarted","Data":"6da06be3b4ab06a51a62ca662f4af0988240f5443b97dcb52c696ef18aa3a0c4"} Jul 14 11:38:52.849784 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:52.849758 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp"] Jul 14 11:38:52.852928 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:38:52.852904 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bc14e0_adc9_429f_a2c6_2f6c1516b8b0.slice/crio-582ab9f048f11747a5dbec1e858da228f477227df9b6e78094c2d1cecb268114 WatchSource:0}: Error finding container 582ab9f048f11747a5dbec1e858da228f477227df9b6e78094c2d1cecb268114: Status 404 returned error can't find the container with id 582ab9f048f11747a5dbec1e858da228f477227df9b6e78094c2d1cecb268114 Jul 14 11:38:53.167761 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.167735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:38:53.167871 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.167778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:53.170898 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.170857 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 11:38:53.171061 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.171043 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 11:38:53.179038 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:53.178985 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 14 11:38:53.179154 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:53.179048 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs podName:eeb5ccae-f66d-42fe-bfca-8016ee9904f8 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:57.17902825 +0000 UTC m=+130.254882821 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs") pod "network-metrics-daemon-t2c79" (UID: "eeb5ccae-f66d-42fe-bfca-8016ee9904f8") : secret "metrics-daemon-secret" not found Jul 14 11:38:53.181091 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.181060 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc-original-pull-secret\") pod \"global-pull-secret-syncer-xkhkk\" (UID: \"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc\") " pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:53.268253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.268211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:53.268377 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.268273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:53.268377 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.268329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:53.268490 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:53.268438 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 11:38:53.268490 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:53.268450 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:55.268436674 +0000 UTC m=+68.344291240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : configmap references non-existent config key: service-ca.crt Jul 14 11:38:53.268592 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:53.268509 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:55.268491368 +0000 UTC m=+68.344345939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : secret "router-metrics-certs-default" not found Jul 14 11:38:53.271619 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.271595 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 11:38:53.281384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.281354 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 11:38:53.291545 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.291519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbpt\" (UniqueName: \"kubernetes.io/projected/1c308823-050d-41d1-8aac-6cd004f01c3c-kube-api-access-rwbpt\") pod \"network-check-target-js79l\" (UID: \"1c308823-050d-41d1-8aac-6cd004f01c3c\") " pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:53.383954 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.383874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkhkk" Jul 14 11:38:53.393967 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.393939 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9256t\"" Jul 14 11:38:53.401749 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.401728 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:53.536967 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.536933 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xkhkk"] Jul 14 11:38:53.539430 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:38:53.539403 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ec8cdd_c0b1_4e3b_9792_a9bbcdbc75fc.slice/crio-00afac7f236a48b2a182f3412053958a715e84e6eff868d48f783225ebf5c706 WatchSource:0}: Error finding container 00afac7f236a48b2a182f3412053958a715e84e6eff868d48f783225ebf5c706: Status 404 returned error can't find the container with id 00afac7f236a48b2a182f3412053958a715e84e6eff868d48f783225ebf5c706 Jul 14 11:38:53.563569 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.563547 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-js79l"] Jul 14 11:38:53.565957 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:38:53.565936 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c308823_050d_41d1_8aac_6cd004f01c3c.slice/crio-50e37ed37a1d5c749edfafa391cd223dfc43e156cec9995a980924b5232c12d1 WatchSource:0}: Error finding container 50e37ed37a1d5c749edfafa391cd223dfc43e156cec9995a980924b5232c12d1: Status 404 returned error can't find the container with id 50e37ed37a1d5c749edfafa391cd223dfc43e156cec9995a980924b5232c12d1 Jul 14 11:38:53.672106 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.672004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:38:53.672263 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:53.672197 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 11:38:53.672401 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:53.672271 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls podName:ec672a19-4fb9-410f-b2c5-7b6ba92830e4 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:25.672250844 +0000 UTC m=+98.748105427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls") pod "dns-default-246h5" (UID: "ec672a19-4fb9-410f-b2c5-7b6ba92830e4") : secret "dns-default-metrics-tls" not found Jul 14 11:38:53.793837 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.793801 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-js79l" event={"ID":"1c308823-050d-41d1-8aac-6cd004f01c3c","Type":"ContainerStarted","Data":"50e37ed37a1d5c749edfafa391cd223dfc43e156cec9995a980924b5232c12d1"} Jul 14 11:38:53.795141 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.795109 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xkhkk" event={"ID":"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc","Type":"ContainerStarted","Data":"00afac7f236a48b2a182f3412053958a715e84e6eff868d48f783225ebf5c706"} Jul 14 11:38:53.796917 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.796836 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q" event={"ID":"91564ec5-d937-4fb6-a637-04f8a18975bd","Type":"ContainerStarted","Data":"952abdf127df5f87a9220602cd391464b1daef60fc52e7e3993bd50b432e1621"} Jul 14 11:38:53.798254 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.798212 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" event={"ID":"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0","Type":"ContainerStarted","Data":"582ab9f048f11747a5dbec1e858da228f477227df9b6e78094c2d1cecb268114"} Jul 14 11:38:53.817733 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:53.817685 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lnt4q" podStartSLOduration=1.646345342 podStartE2EDuration="2.817666128s" totalImagesPullingTime="1.171320786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:53.817451031 +0000 UTC m=+66.893305620" watchObservedRunningTime="2026-07-14 11:38:53.817666128 +0000 UTC m=+66.893520718" Jul 14 11:38:54.803987 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:54.803281 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/0.log" Jul 14 11:38:54.803987 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:54.803331 2562 generic.go:356] "Generic (PLEG): container finished" podID="a500e755-d871-4992-9b39-18ef1f7e9330" containerID="6a20abafc82cd5ba58629a88c7b04fb5fcbd246ba0c11f3fb6ccad327ee80a82" exitCode=255 Jul 14 11:38:54.803987 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:54.803900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-k5nq8" event={"ID":"a500e755-d871-4992-9b39-18ef1f7e9330","Type":"ContainerDied","Data":"6a20abafc82cd5ba58629a88c7b04fb5fcbd246ba0c11f3fb6ccad327ee80a82"} Jul 14 11:38:54.803987 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:54.803913 2562 scope.go:117] "RemoveContainer" containerID="6a20abafc82cd5ba58629a88c7b04fb5fcbd246ba0c11f3fb6ccad327ee80a82" Jul 14 11:38:55.285885 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.285750 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:55.285885 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.285840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:55.286134 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:55.286060 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:59.286042486 +0000 UTC m=+72.361897066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : configmap references non-existent config key: service-ca.crt Jul 14 11:38:55.286205 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:55.286181 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 11:38:55.286256 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:55.286236 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:38:59.286219276 +0000 UTC m=+72.362073855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : secret "router-metrics-certs-default" not found Jul 14 11:38:55.455227 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.455102 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d"] Jul 14 11:38:55.457215 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.457185 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d" Jul 14 11:38:55.460316 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.460289 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8tcmb\"" Jul 14 11:38:55.468774 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.468746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d"] Jul 14 11:38:55.487989 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.487961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lp2r\" (UniqueName: \"kubernetes.io/projected/4f42efff-d6f9-4141-b0d9-73a2fb27b405-kube-api-access-7lp2r\") pod \"network-check-source-869dcfcdcd-jb84d\" (UID: \"4f42efff-d6f9-4141-b0d9-73a2fb27b405\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d" Jul 14 11:38:55.588790 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.588763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lp2r\" (UniqueName: \"kubernetes.io/projected/4f42efff-d6f9-4141-b0d9-73a2fb27b405-kube-api-access-7lp2r\") pod \"network-check-source-869dcfcdcd-jb84d\" (UID: \"4f42efff-d6f9-4141-b0d9-73a2fb27b405\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d" Jul 14 11:38:55.600291 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.600260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lp2r\" (UniqueName: \"kubernetes.io/projected/4f42efff-d6f9-4141-b0d9-73a2fb27b405-kube-api-access-7lp2r\") pod \"network-check-source-869dcfcdcd-jb84d\" (UID: \"4f42efff-d6f9-4141-b0d9-73a2fb27b405\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d" Jul 14 11:38:55.769487 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.769453 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d" Jul 14 11:38:55.811575 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.811536 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" event={"ID":"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0","Type":"ContainerStarted","Data":"2a42061192c92ef6fbfc2976b0c0635fa42b02d64da88f26d104cafbbde94559"} Jul 14 11:38:55.815093 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.815051 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:38:55.815580 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.815562 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/0.log" Jul 14 11:38:55.815676 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.815597 2562 generic.go:356] "Generic (PLEG): container finished" podID="a500e755-d871-4992-9b39-18ef1f7e9330" containerID="330247ca6d738f7405ef17f9ea3a3786839837f049e713f6e7dc71eba99fa08e" exitCode=255 Jul 14 11:38:55.815676 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.815659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-k5nq8" event={"ID":"a500e755-d871-4992-9b39-18ef1f7e9330","Type":"ContainerDied","Data":"330247ca6d738f7405ef17f9ea3a3786839837f049e713f6e7dc71eba99fa08e"} Jul 14 11:38:55.815778 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.815700 2562 scope.go:117] "RemoveContainer" containerID="6a20abafc82cd5ba58629a88c7b04fb5fcbd246ba0c11f3fb6ccad327ee80a82" Jul 14 11:38:55.816054 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.816037 2562 scope.go:117] "RemoveContainer" containerID="330247ca6d738f7405ef17f9ea3a3786839837f049e713f6e7dc71eba99fa08e" Jul 14 11:38:55.816242 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:55.816211 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-k5nq8_openshift-console-operator(a500e755-d871-4992-9b39-18ef1f7e9330)\"" pod="openshift-console-operator/console-operator-7866497494-k5nq8" podUID="a500e755-d871-4992-9b39-18ef1f7e9330" Jul 14 11:38:55.832922 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.832512 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" podStartSLOduration=1.454076159 podStartE2EDuration="3.832491609s" totalImagesPullingTime="2.37841545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:55.832314887 +0000 UTC m=+68.908169475" watchObservedRunningTime="2026-07-14 11:38:55.832491609 +0000 UTC m=+68.908346198" Jul 14 11:38:55.936255 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:55.936229 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d"] Jul 14 11:38:56.176903 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.176830 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w9qsj"] Jul 14 11:38:56.187490 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.187466 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.190741 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.190723 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 11:38:56.190971 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.190950 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 11:38:56.191529 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.191508 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 11:38:56.191638 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.191562 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 11:38:56.191638 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.191562 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mf86b\"" Jul 14 11:38:56.194700 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.194594 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9qsj"] Jul 14 11:38:56.295148 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.295103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.295309 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.295214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/123b2e42-1ec7-459b-ae70-872058d5465e-crio-socket\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.295309 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.295241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzd4v\" (UniqueName: \"kubernetes.io/projected/123b2e42-1ec7-459b-ae70-872058d5465e-kube-api-access-kzd4v\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.295309 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.295267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/123b2e42-1ec7-459b-ae70-872058d5465e-data-volume\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.295309 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.295294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.396479 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.396439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/123b2e42-1ec7-459b-ae70-872058d5465e-data-volume\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.396631 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.396488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.396631 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.396555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.396631 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.396626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/123b2e42-1ec7-459b-ae70-872058d5465e-crio-socket\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.396928 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.396648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzd4v\" (UniqueName: \"kubernetes.io/projected/123b2e42-1ec7-459b-ae70-872058d5465e-kube-api-access-kzd4v\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.396928 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:56.396803 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 11:38:56.396928 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.396829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/123b2e42-1ec7-459b-ae70-872058d5465e-data-volume\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.396928 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.396851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/123b2e42-1ec7-459b-ae70-872058d5465e-crio-socket\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.396928 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:56.396876 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:38:56.896854435 +0000 UTC m=+69.972709021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : secret "insights-runtime-extractor-tls" not found Jul 14 11:38:56.396928 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:56.396903 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 11:38:56.397233 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:56.396944 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:38:56.8969322 +0000 UTC m=+69.972786766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : configmap "kube-rbac-proxy" not found Jul 14 11:38:56.409213 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.409188 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzd4v\" (UniqueName: \"kubernetes.io/projected/123b2e42-1ec7-459b-ae70-872058d5465e-kube-api-access-kzd4v\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.820281 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.820245 2562 scope.go:117] "RemoveContainer" containerID="330247ca6d738f7405ef17f9ea3a3786839837f049e713f6e7dc71eba99fa08e" Jul 14 11:38:56.820671 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:56.820429 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-k5nq8_openshift-console-operator(a500e755-d871-4992-9b39-18ef1f7e9330)\"" pod="openshift-console-operator/console-operator-7866497494-k5nq8" podUID="a500e755-d871-4992-9b39-18ef1f7e9330" Jul 14 11:38:56.901026 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.900906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.901197 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:56.901056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:56.901197 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:56.901151 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 11:38:56.901323 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:56.901201 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 11:38:56.901323 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:56.901204 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:38:57.901186471 +0000 UTC m=+70.977041059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : configmap "kube-rbac-proxy" not found Jul 14 11:38:56.901323 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:56.901264 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:38:57.901245824 +0000 UTC m=+70.977100402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : secret "insights-runtime-extractor-tls" not found Jul 14 11:38:56.975720 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:38:56.975696 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f42efff_d6f9_4141_b0d9_73a2fb27b405.slice/crio-a4e2ed523c1884d8cc010827bf173082ae113871b21d0d4667e8cedf0234a9df WatchSource:0}: Error finding container a4e2ed523c1884d8cc010827bf173082ae113871b21d0d4667e8cedf0234a9df: Status 404 returned error can't find the container with id a4e2ed523c1884d8cc010827bf173082ae113871b21d0d4667e8cedf0234a9df Jul 14 11:38:57.824066 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:57.824026 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d" event={"ID":"4f42efff-d6f9-4141-b0d9-73a2fb27b405","Type":"ContainerStarted","Data":"a4e2ed523c1884d8cc010827bf173082ae113871b21d0d4667e8cedf0234a9df"} Jul 14 11:38:57.825855 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:57.825835 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:38:57.909458 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:57.909418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:57.909637 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:57.909544 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 11:38:57.909637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:57.909606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:57.909742 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:57.909685 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:38:59.909669957 +0000 UTC m=+72.985524524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : configmap "kube-rbac-proxy" not found Jul 14 11:38:57.909802 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:57.909761 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 11:38:57.909851 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:57.909827 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:38:59.909808027 +0000 UTC m=+72.985662594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : secret "insights-runtime-extractor-tls" not found Jul 14 11:38:58.450546 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:58.450518 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tfd9j_eed3a9e7-5412-4606-b01a-a852629061a5/dns-node-resolver/0.log" Jul 14 11:38:58.831296 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:58.831248 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-js79l" event={"ID":"1c308823-050d-41d1-8aac-6cd004f01c3c","Type":"ContainerStarted","Data":"44cb20fcf438af60731452b4bb349a0d2ccf24a588c254bc58e47c897757f239"} Jul 14 11:38:58.860119 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:58.860049 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-js79l" podStartSLOduration=65.696223071 podStartE2EDuration="1m10.860030135s" totalImagesPullingTime="5.163807064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:58.857873034 +0000 UTC m=+71.933727624" watchObservedRunningTime="2026-07-14 11:38:58.860030135 +0000 UTC m=+71.935884703" Jul 14 11:38:59.048104 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.048021 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dgl7z_1ff938fe-df60-4fb5-b558-e7832e86ff4b/node-ca/0.log" Jul 14 11:38:59.321518 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.321453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:59.321518 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.321503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:38:59.321781 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:59.321586 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 11:38:59.321781 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:59.321620 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:07.321604697 +0000 UTC m=+80.397459277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : configmap references non-existent config key: service-ca.crt Jul 14 11:38:59.321781 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:59.321635 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:07.321628957 +0000 UTC m=+80.397483524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : secret "router-metrics-certs-default" not found Jul 14 11:38:59.654850 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.654765 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5bdxl_96d809e2-9c08-4cb9-8a57-2c3aef340d37/serve-healthcheck-canary/0.log" Jul 14 11:38:59.837686 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.837656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xkhkk" event={"ID":"03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc","Type":"ContainerStarted","Data":"b486b43a7971bfeee92c58380882480006e6f1e0eb6e84ac221f1bc9f0508d11"} Jul 14 11:38:59.839126 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.839102 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d" event={"ID":"4f42efff-d6f9-4141-b0d9-73a2fb27b405","Type":"ContainerStarted","Data":"cbd1316305573b84717a240a51a633ea2ebda0167ea0774823b282087d6bfbdb"} Jul 14 11:38:59.839226 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.839210 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:38:59.866956 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.866911 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xkhkk" podStartSLOduration=66.676759391 podStartE2EDuration="1m11.866895305s" totalImagesPullingTime="5.190135914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:59.866449722 +0000 UTC m=+72.942304310" watchObservedRunningTime="2026-07-14 11:38:59.866895305 +0000 UTC m=+72.942749895" Jul 14 11:38:59.896568 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.896526 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jb84d" podStartSLOduration=3.141742804 podStartE2EDuration="4.896512363s" totalImagesPullingTime="1.754769559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:38:59.896007272 +0000 UTC m=+72.971861861" watchObservedRunningTime="2026-07-14 11:38:59.896512363 +0000 UTC m=+72.972366951" Jul 14 11:38:59.925156 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.925100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:59.925253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:38:59.925164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:38:59.925253 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:59.925234 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 11:38:59.925353 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:59.925288 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 11:38:59.925353 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:59.925296 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:39:03.925278708 +0000 UTC m=+77.001133312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : configmap "kube-rbac-proxy" not found Jul 14 11:38:59.925353 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:38:59.925353 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:39:03.925337522 +0000 UTC m=+77.001192095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : secret "insights-runtime-extractor-tls" not found Jul 14 11:39:01.485810 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.485778 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-shrp6"] Jul 14 11:39:01.487834 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.487815 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.490886 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.490866 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 14 11:39:01.492273 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.492255 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 14 11:39:01.492349 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.492272 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 14 11:39:01.492349 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.492283 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dqdkh\"" Jul 14 11:39:01.492349 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.492303 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 14 11:39:01.503424 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.503403 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-shrp6"] Jul 14 11:39:01.538637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.538608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vm8\" (UniqueName: \"kubernetes.io/projected/6970b667-9b32-45a4-aebb-376184ec17e7-kube-api-access-79vm8\") pod \"service-ca-79f68b9f4f-shrp6\" (UID: \"6970b667-9b32-45a4-aebb-376184ec17e7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.538727 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.538645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6970b667-9b32-45a4-aebb-376184ec17e7-signing-cabundle\") pod \"service-ca-79f68b9f4f-shrp6\" (UID: \"6970b667-9b32-45a4-aebb-376184ec17e7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.538727 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.538707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6970b667-9b32-45a4-aebb-376184ec17e7-signing-key\") pod \"service-ca-79f68b9f4f-shrp6\" (UID: \"6970b667-9b32-45a4-aebb-376184ec17e7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.639478 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.639452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vm8\" (UniqueName: \"kubernetes.io/projected/6970b667-9b32-45a4-aebb-376184ec17e7-kube-api-access-79vm8\") pod \"service-ca-79f68b9f4f-shrp6\" (UID: \"6970b667-9b32-45a4-aebb-376184ec17e7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.639566 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.639485 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6970b667-9b32-45a4-aebb-376184ec17e7-signing-cabundle\") pod \"service-ca-79f68b9f4f-shrp6\" (UID: \"6970b667-9b32-45a4-aebb-376184ec17e7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.639566 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.639549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6970b667-9b32-45a4-aebb-376184ec17e7-signing-key\") pod \"service-ca-79f68b9f4f-shrp6\" (UID: \"6970b667-9b32-45a4-aebb-376184ec17e7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.640074 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.640053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6970b667-9b32-45a4-aebb-376184ec17e7-signing-cabundle\") pod \"service-ca-79f68b9f4f-shrp6\" (UID: \"6970b667-9b32-45a4-aebb-376184ec17e7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.641727 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.641707 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6970b667-9b32-45a4-aebb-376184ec17e7-signing-key\") pod \"service-ca-79f68b9f4f-shrp6\" (UID: \"6970b667-9b32-45a4-aebb-376184ec17e7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.652042 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.652017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vm8\" (UniqueName: \"kubernetes.io/projected/6970b667-9b32-45a4-aebb-376184ec17e7-kube-api-access-79vm8\") pod \"service-ca-79f68b9f4f-shrp6\" (UID: \"6970b667-9b32-45a4-aebb-376184ec17e7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.795537 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.795492 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" Jul 14 11:39:01.914545 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.914512 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-shrp6"] Jul 14 11:39:01.917483 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:01.917463 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6970b667_9b32_45a4_aebb_376184ec17e7.slice/crio-0ac3d069c88a82f525bff0d68896bd18f757d9424bc5ec6427d7dfc6434ac102 WatchSource:0}: Error finding container 0ac3d069c88a82f525bff0d68896bd18f757d9424bc5ec6427d7dfc6434ac102: Status 404 returned error can't find the container with id 0ac3d069c88a82f525bff0d68896bd18f757d9424bc5ec6427d7dfc6434ac102 Jul 14 11:39:01.938630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.938607 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:39:01.938712 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.938653 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:39:01.938938 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:01.938927 2562 scope.go:117] "RemoveContainer" containerID="330247ca6d738f7405ef17f9ea3a3786839837f049e713f6e7dc71eba99fa08e" Jul 14 11:39:01.939103 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:01.939062 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-k5nq8_openshift-console-operator(a500e755-d871-4992-9b39-18ef1f7e9330)\"" pod="openshift-console-operator/console-operator-7866497494-k5nq8" podUID="a500e755-d871-4992-9b39-18ef1f7e9330" Jul 14 11:39:02.852698 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:02.852661 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" event={"ID":"6970b667-9b32-45a4-aebb-376184ec17e7","Type":"ContainerStarted","Data":"0ac3d069c88a82f525bff0d68896bd18f757d9424bc5ec6427d7dfc6434ac102"} Jul 14 11:39:03.959311 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:03.959276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:39:03.959677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:03.959333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:39:03.959677 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:03.959395 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 11:39:03.959677 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:03.959477 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:39:11.959461305 +0000 UTC m=+85.035315892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : configmap "kube-rbac-proxy" not found Jul 14 11:39:03.959677 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:03.959526 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 11:39:03.959677 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:03.959596 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:39:11.959578168 +0000 UTC m=+85.035432745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : secret "insights-runtime-extractor-tls" not found Jul 14 11:39:04.861852 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:04.861820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" event={"ID":"6970b667-9b32-45a4-aebb-376184ec17e7","Type":"ContainerStarted","Data":"c5c3bc07c96e3194c6fc0765cfa7b09626b912990b375627640eb0044e4032a4"} Jul 14 11:39:04.927535 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:04.927481 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-shrp6" podStartSLOduration=2.042543256 podStartE2EDuration="3.927466549s" totalImagesPullingTime="1.884923293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:04.926336666 +0000 UTC m=+78.002191248" watchObservedRunningTime="2026-07-14 11:39:04.927466549 +0000 UTC m=+78.003321197" Jul 14 11:39:07.385415 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:07.385380 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:07.385875 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:07.385427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:07.385875 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:07.385540 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 11:39:07.385875 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:07.385556 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:23.385533717 +0000 UTC m=+96.461388287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : configmap references non-existent config key: service-ca.crt Jul 14 11:39:07.385875 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:07.385606 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs podName:3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:23.385588704 +0000 UTC m=+96.461443284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs") pod "router-default-7894875846-lcf2d" (UID: "3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4") : secret "router-metrics-certs-default" not found Jul 14 11:39:12.021384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:12.021278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:39:12.021384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:12.021322 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:39:12.021876 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:12.021408 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 11:39:12.021876 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:12.021456 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls podName:123b2e42-1ec7-459b-ae70-872058d5465e nodeName:}" failed. No retries permitted until 2026-07-14 11:39:28.021443497 +0000 UTC m=+101.097298067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9qsj" (UID: "123b2e42-1ec7-459b-ae70-872058d5465e") : secret "insights-runtime-extractor-tls" not found Jul 14 11:39:12.021876 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:12.021845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/123b2e42-1ec7-459b-ae70-872058d5465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:39:17.474905 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:17.474872 2562 scope.go:117] "RemoveContainer" containerID="330247ca6d738f7405ef17f9ea3a3786839837f049e713f6e7dc71eba99fa08e" Jul 14 11:39:17.901757 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:17.901680 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:39:17.901930 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:17.901766 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-k5nq8" event={"ID":"a500e755-d871-4992-9b39-18ef1f7e9330","Type":"ContainerStarted","Data":"a26d1f3030f0caddef122733a13aab6988be79721af48bac1b6bfe76e6a58592"} Jul 14 11:39:17.902327 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:17.902265 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:39:17.908189 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:17.908165 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-k5nq8" Jul 14 11:39:17.926270 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:17.926229 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-k5nq8" podStartSLOduration=24.850557486 podStartE2EDuration="26.926215571s" totalImagesPullingTime="2.075658085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:17.92503286 +0000 UTC m=+91.000887461" watchObservedRunningTime="2026-07-14 11:39:17.926215571 +0000 UTC m=+91.002070158" Jul 14 11:39:23.396188 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:23.396156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:23.396542 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:23.396226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:23.396722 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:23.396700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-service-ca-bundle\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:23.398369 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:23.398346 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4-metrics-certs\") pod \"router-default-7894875846-lcf2d\" (UID: \"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4\") " pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:23.433382 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:23.433358 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:23.553733 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:23.553702 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7894875846-lcf2d"] Jul 14 11:39:23.556524 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:23.556499 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0972d2_99aa_4b5a_9ba0_a3d64289f8f4.slice/crio-557327c25eb7a86d8fdd30ffae52fe99353e5ebb71e3c25db0e2e5318e84cb6a WatchSource:0}: Error finding container 557327c25eb7a86d8fdd30ffae52fe99353e5ebb71e3c25db0e2e5318e84cb6a: Status 404 returned error can't find the container with id 557327c25eb7a86d8fdd30ffae52fe99353e5ebb71e3c25db0e2e5318e84cb6a Jul 14 11:39:23.927560 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:23.927526 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7894875846-lcf2d" event={"ID":"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4","Type":"ContainerStarted","Data":"4570b615337299fd8e0d957db1395e8fad47821127b5b1e45ec7745a568721fd"} Jul 14 11:39:23.927722 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:23.927571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7894875846-lcf2d" event={"ID":"3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4","Type":"ContainerStarted","Data":"557327c25eb7a86d8fdd30ffae52fe99353e5ebb71e3c25db0e2e5318e84cb6a"} Jul 14 11:39:23.952410 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:23.952370 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7894875846-lcf2d" podStartSLOduration=32.952355843 podStartE2EDuration="32.952355843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:23.950404401 +0000 UTC m=+97.026258988" watchObservedRunningTime="2026-07-14 11:39:23.952355843 +0000 UTC m=+97.028210429" Jul 14 11:39:24.433969 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:24.433934 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:24.436576 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:24.436552 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:24.930965 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:24.930930 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:24.932015 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:24.931995 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7894875846-lcf2d" Jul 14 11:39:25.712486 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:25.712457 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:39:25.714664 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:25.714643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec672a19-4fb9-410f-b2c5-7b6ba92830e4-metrics-tls\") pod \"dns-default-246h5\" (UID: \"ec672a19-4fb9-410f-b2c5-7b6ba92830e4\") " pod="openshift-dns/dns-default-246h5" Jul 14 11:39:25.724542 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:25.724522 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9dtrp\"" Jul 14 11:39:25.731919 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:25.731905 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-246h5" Jul 14 11:39:25.866454 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:25.866429 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-246h5"] Jul 14 11:39:25.868871 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:25.868840 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec672a19_4fb9_410f_b2c5_7b6ba92830e4.slice/crio-565c8118313624af22dd6d42b2979fcc98fe7c003b6f0ac7e47dcb231b083893 WatchSource:0}: Error finding container 565c8118313624af22dd6d42b2979fcc98fe7c003b6f0ac7e47dcb231b083893: Status 404 returned error can't find the container with id 565c8118313624af22dd6d42b2979fcc98fe7c003b6f0ac7e47dcb231b083893 Jul 14 11:39:25.934872 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:25.934841 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-246h5" event={"ID":"ec672a19-4fb9-410f-b2c5-7b6ba92830e4","Type":"ContainerStarted","Data":"565c8118313624af22dd6d42b2979fcc98fe7c003b6f0ac7e47dcb231b083893"} Jul 14 11:39:28.029521 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:28.029488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:39:28.031635 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:28.031612 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/123b2e42-1ec7-459b-ae70-872058d5465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9qsj\" (UID: \"123b2e42-1ec7-459b-ae70-872058d5465e\") " pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:39:28.300171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:28.300146 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9qsj" Jul 14 11:39:28.453387 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:28.453356 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9qsj"] Jul 14 11:39:28.456803 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:28.456781 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123b2e42_1ec7_459b_ae70_872058d5465e.slice/crio-bc656d586803cdda42f53a3c871a71a6845d505a0edd54020ff263d3d479a9ec WatchSource:0}: Error finding container bc656d586803cdda42f53a3c871a71a6845d505a0edd54020ff263d3d479a9ec: Status 404 returned error can't find the container with id bc656d586803cdda42f53a3c871a71a6845d505a0edd54020ff263d3d479a9ec Jul 14 11:39:28.944682 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:28.944645 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9qsj" event={"ID":"123b2e42-1ec7-459b-ae70-872058d5465e","Type":"ContainerStarted","Data":"bc656d586803cdda42f53a3c871a71a6845d505a0edd54020ff263d3d479a9ec"} Jul 14 11:39:28.946114 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:28.946067 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-246h5" event={"ID":"ec672a19-4fb9-410f-b2c5-7b6ba92830e4","Type":"ContainerStarted","Data":"97de20ce90dcfe4b9eaf71f4a20007a762ff5873ff8372f166a5a0e5c879f179"} Jul 14 11:39:28.946194 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:28.946137 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-246h5" event={"ID":"ec672a19-4fb9-410f-b2c5-7b6ba92830e4","Type":"ContainerStarted","Data":"00e018e7e5fa0e8e2c53c7cc1b4603a333a9710cef6823c64873c2d0a9a586ab"} Jul 14 11:39:28.946194 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:28.946182 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-246h5" Jul 14 11:39:28.982977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:28.982933 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-246h5" podStartSLOduration=65.732126706 podStartE2EDuration="1m7.982920129s" totalImagesPullingTime="2.250793423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:28.982402365 +0000 UTC m=+102.058256954" watchObservedRunningTime="2026-07-14 11:39:28.982920129 +0000 UTC m=+102.058774716" Jul 14 11:39:30.846661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:30.846628 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-js79l" Jul 14 11:39:31.605708 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.605675 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-ccmt2"] Jul 14 11:39:31.616234 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.616211 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" Jul 14 11:39:31.621316 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.620830 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 14 11:39:31.621316 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.620923 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 14 11:39:31.621487 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.621386 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lbsw7\"" Jul 14 11:39:31.622043 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.622022 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-ccmt2"] Jul 14 11:39:31.708431 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.708399 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w"] Jul 14 11:39:31.711781 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.711758 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" Jul 14 11:39:31.714407 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.714386 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bfb7987c8-qh4pl"] Jul 14 11:39:31.714992 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.714968 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8cclr\"" Jul 14 11:39:31.716251 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.716230 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 14 11:39:31.717118 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.717093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.719870 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.719854 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 11:39:31.720339 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.720290 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 11:39:31.721340 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.720934 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ggxrg\"" Jul 14 11:39:31.721340 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.720987 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 11:39:31.721340 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.721009 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 11:39:31.727278 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.721652 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 11:39:31.727278 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.721866 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 11:39:31.727278 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.722119 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 11:39:31.730151 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.730038 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w"] Jul 14 11:39:31.731658 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.731638 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 11:39:31.739322 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.739300 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bfb7987c8-qh4pl"] Jul 14 11:39:31.758321 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.758286 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ab73933c-fedc-474f-b0fa-a68e49f0f6ea-nginx-conf\") pod \"networking-console-plugin-6cf856758c-ccmt2\" (UID: \"ab73933c-fedc-474f-b0fa-a68e49f0f6ea\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" Jul 14 11:39:31.758430 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.758339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab73933c-fedc-474f-b0fa-a68e49f0f6ea-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ccmt2\" (UID: \"ab73933c-fedc-474f-b0fa-a68e49f0f6ea\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" Jul 14 11:39:31.765032 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.765008 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c46ccf68d-5rwrx"] Jul 14 11:39:31.767937 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.767918 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.771237 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.771219 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zdzfm\"" Jul 14 11:39:31.771522 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.771493 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 11:39:31.771888 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.771856 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 11:39:31.772010 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.771886 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 11:39:31.778171 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.778154 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 11:39:31.783036 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.783018 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c46ccf68d-5rwrx"] Jul 14 11:39:31.860169 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860065 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-oauth-config\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.860169 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860130 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-config\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ef2bb0-98f3-4283-b43a-7e51484a79c7-ca-trust-extracted\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860200 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ef2bb0-98f3-4283-b43a-7e51484a79c7-trusted-ca\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ab73933c-fedc-474f-b0fa-a68e49f0f6ea-nginx-conf\") pod \"networking-console-plugin-6cf856758c-ccmt2\" (UID: \"ab73933c-fedc-474f-b0fa-a68e49f0f6ea\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860254 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr6f9\" (UniqueName: \"kubernetes.io/projected/86ef2bb0-98f3-4283-b43a-7e51484a79c7-kube-api-access-qr6f9\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab73933c-fedc-474f-b0fa-a68e49f0f6ea-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ccmt2\" (UID: \"ab73933c-fedc-474f-b0fa-a68e49f0f6ea\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860288 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ef2bb0-98f3-4283-b43a-7e51484a79c7-installation-pull-secrets\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ef2bb0-98f3-4283-b43a-7e51484a79c7-registry-certificates\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-service-ca\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860391 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c8ad185f-7916-4073-baf7-1db23a65afd2-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8dk7w\" (UID: \"c8ad185f-7916-4073-baf7-1db23a65afd2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-oauth-serving-cert\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860467 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ef2bb0-98f3-4283-b43a-7e51484a79c7-registry-tls\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860506 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ef2bb0-98f3-4283-b43a-7e51484a79c7-bound-sa-token\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.860637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-serving-cert\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.861259 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860583 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9mt\" (UniqueName: \"kubernetes.io/projected/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-kube-api-access-rn9mt\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.861259 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860623 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ef2bb0-98f3-4283-b43a-7e51484a79c7-image-registry-private-configuration\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.861259 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860651 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-trusted-ca-bundle\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.861259 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.860939 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ab73933c-fedc-474f-b0fa-a68e49f0f6ea-nginx-conf\") pod \"networking-console-plugin-6cf856758c-ccmt2\" (UID: \"ab73933c-fedc-474f-b0fa-a68e49f0f6ea\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" Jul 14 11:39:31.862775 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.862749 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab73933c-fedc-474f-b0fa-a68e49f0f6ea-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-ccmt2\" (UID: \"ab73933c-fedc-474f-b0fa-a68e49f0f6ea\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" Jul 14 11:39:31.927461 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.927430 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" Jul 14 11:39:31.961406 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961384 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr6f9\" (UniqueName: \"kubernetes.io/projected/86ef2bb0-98f3-4283-b43a-7e51484a79c7-kube-api-access-qr6f9\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.961569 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ef2bb0-98f3-4283-b43a-7e51484a79c7-installation-pull-secrets\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.961569 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ef2bb0-98f3-4283-b43a-7e51484a79c7-registry-certificates\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.961569 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-service-ca\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.961569 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c8ad185f-7916-4073-baf7-1db23a65afd2-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8dk7w\" (UID: \"c8ad185f-7916-4073-baf7-1db23a65afd2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" Jul 14 11:39:31.961569 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-oauth-serving-cert\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961596 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ef2bb0-98f3-4283-b43a-7e51484a79c7-registry-tls\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ef2bb0-98f3-4283-b43a-7e51484a79c7-bound-sa-token\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-serving-cert\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9mt\" (UniqueName: \"kubernetes.io/projected/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-kube-api-access-rn9mt\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ef2bb0-98f3-4283-b43a-7e51484a79c7-image-registry-private-configuration\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-trusted-ca-bundle\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-oauth-config\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961837 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-config\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ef2bb0-98f3-4283-b43a-7e51484a79c7-ca-trust-extracted\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.962158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.961887 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ef2bb0-98f3-4283-b43a-7e51484a79c7-trusted-ca\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.962747 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.962251 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-service-ca\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.962747 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.962317 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ef2bb0-98f3-4283-b43a-7e51484a79c7-registry-certificates\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.962747 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.962715 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-oauth-serving-cert\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.962902 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.962879 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ef2bb0-98f3-4283-b43a-7e51484a79c7-trusted-ca\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.963598 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.963309 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ef2bb0-98f3-4283-b43a-7e51484a79c7-ca-trust-extracted\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.963598 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.963474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-config\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.963598 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.963577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-trusted-ca-bundle\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.964828 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.964798 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c8ad185f-7916-4073-baf7-1db23a65afd2-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-8dk7w\" (UID: \"c8ad185f-7916-4073-baf7-1db23a65afd2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" Jul 14 11:39:31.965096 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.965032 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-serving-cert\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.965391 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.965368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-oauth-config\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.965464 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.965442 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ef2bb0-98f3-4283-b43a-7e51484a79c7-installation-pull-secrets\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.965786 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.965765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ef2bb0-98f3-4283-b43a-7e51484a79c7-image-registry-private-configuration\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.966004 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.965984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ef2bb0-98f3-4283-b43a-7e51484a79c7-registry-tls\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.972743 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.972719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9mt\" (UniqueName: \"kubernetes.io/projected/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-kube-api-access-rn9mt\") pod \"console-6bfb7987c8-qh4pl\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:31.973556 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.973534 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ef2bb0-98f3-4283-b43a-7e51484a79c7-bound-sa-token\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:31.974002 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:31.973982 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr6f9\" (UniqueName: \"kubernetes.io/projected/86ef2bb0-98f3-4283-b43a-7e51484a79c7-kube-api-access-qr6f9\") pod \"image-registry-c46ccf68d-5rwrx\" (UID: \"86ef2bb0-98f3-4283-b43a-7e51484a79c7\") " pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:32.028746 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.028719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" Jul 14 11:39:32.035875 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.035853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:32.078003 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.077980 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:32.370747 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.370701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-ccmt2"] Jul 14 11:39:32.372636 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:32.372612 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab73933c_fedc_474f_b0fa_a68e49f0f6ea.slice/crio-9ff23a582d962dba04d65f1e41f386e1e390ff15d9d15e23d8eb5928a17cf45f WatchSource:0}: Error finding container 9ff23a582d962dba04d65f1e41f386e1e390ff15d9d15e23d8eb5928a17cf45f: Status 404 returned error can't find the container with id 9ff23a582d962dba04d65f1e41f386e1e390ff15d9d15e23d8eb5928a17cf45f Jul 14 11:39:32.374121 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.374062 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c46ccf68d-5rwrx"] Jul 14 11:39:32.375057 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:32.375032 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ef2bb0_98f3_4283_b43a_7e51484a79c7.slice/crio-4db40069cd589704cac54e2d5f847cbf0aed92764ca7aa03eb7af1232fc0fee1 WatchSource:0}: Error finding container 4db40069cd589704cac54e2d5f847cbf0aed92764ca7aa03eb7af1232fc0fee1: Status 404 returned error can't find the container with id 4db40069cd589704cac54e2d5f847cbf0aed92764ca7aa03eb7af1232fc0fee1 Jul 14 11:39:32.392666 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.392645 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w"] Jul 14 11:39:32.408030 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:32.408006 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ad185f_7916_4073_baf7_1db23a65afd2.slice/crio-0bc3c843d3454b51e2a6d5ab37b44fee969b818c18734607a9b5857764d718ac WatchSource:0}: Error finding container 0bc3c843d3454b51e2a6d5ab37b44fee969b818c18734607a9b5857764d718ac: Status 404 returned error can't find the container with id 0bc3c843d3454b51e2a6d5ab37b44fee969b818c18734607a9b5857764d718ac Jul 14 11:39:32.562531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.562507 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bfb7987c8-qh4pl"] Jul 14 11:39:32.564420 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:32.564396 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4e54fb_a1a6_43ec_865c_9994a9ac929e.slice/crio-7e105ce6620eab03c83afa6d5c9fe32d4a33a7b2f941c8abb5922e79bc1262f3 WatchSource:0}: Error finding container 7e105ce6620eab03c83afa6d5c9fe32d4a33a7b2f941c8abb5922e79bc1262f3: Status 404 returned error can't find the container with id 7e105ce6620eab03c83afa6d5c9fe32d4a33a7b2f941c8abb5922e79bc1262f3 Jul 14 11:39:32.963147 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.963110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9qsj" event={"ID":"123b2e42-1ec7-459b-ae70-872058d5465e","Type":"ContainerStarted","Data":"a27ed3881b2a3051022be06ca815f04bc342da4e1560379c401fec02f33ac56a"} Jul 14 11:39:32.963459 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.963188 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9qsj" event={"ID":"123b2e42-1ec7-459b-ae70-872058d5465e","Type":"ContainerStarted","Data":"89ee1d4982319da9a49c2707ecbef3266f57452c22996fdc6fd2c85fa4a907ea"} Jul 14 11:39:32.964663 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.964637 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" event={"ID":"c8ad185f-7916-4073-baf7-1db23a65afd2","Type":"ContainerStarted","Data":"0bc3c843d3454b51e2a6d5ab37b44fee969b818c18734607a9b5857764d718ac"} Jul 14 11:39:32.965838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.965811 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" event={"ID":"ab73933c-fedc-474f-b0fa-a68e49f0f6ea","Type":"ContainerStarted","Data":"9ff23a582d962dba04d65f1e41f386e1e390ff15d9d15e23d8eb5928a17cf45f"} Jul 14 11:39:32.967137 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.967108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfb7987c8-qh4pl" event={"ID":"bc4e54fb-a1a6-43ec-865c-9994a9ac929e","Type":"ContainerStarted","Data":"7e105ce6620eab03c83afa6d5c9fe32d4a33a7b2f941c8abb5922e79bc1262f3"} Jul 14 11:39:32.968944 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.968921 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" event={"ID":"86ef2bb0-98f3-4283-b43a-7e51484a79c7","Type":"ContainerStarted","Data":"b8d355021c71e09cabf3721f75a26f80848269417d20e1d54284708b50316110"} Jul 14 11:39:32.969056 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.968962 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" event={"ID":"86ef2bb0-98f3-4283-b43a-7e51484a79c7","Type":"ContainerStarted","Data":"4db40069cd589704cac54e2d5f847cbf0aed92764ca7aa03eb7af1232fc0fee1"} Jul 14 11:39:32.969141 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.969092 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:32.996382 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:32.996299 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" podStartSLOduration=1.996281311 podStartE2EDuration="1.996281311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:32.994782625 +0000 UTC m=+106.070637219" watchObservedRunningTime="2026-07-14 11:39:32.996281311 +0000 UTC m=+106.072135899" Jul 14 11:39:33.975780 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:33.975739 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" event={"ID":"ab73933c-fedc-474f-b0fa-a68e49f0f6ea","Type":"ContainerStarted","Data":"f19f308986ef07fc34e4fdf27560b4d84a25bb93bdf5f6011e5cd6b87ab7272b"} Jul 14 11:39:33.977810 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:33.977751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" event={"ID":"c8ad185f-7916-4073-baf7-1db23a65afd2","Type":"ContainerStarted","Data":"a41ea1ae184202b2865c2ff142cdf134ac40d6a6864d4e7d23f3a4d3df90ec00"} Jul 14 11:39:33.997407 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:33.997353 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-ccmt2" podStartSLOduration=1.539857069 podStartE2EDuration="2.997333636s" totalImagesPullingTime="1.457476567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:33.996394156 +0000 UTC m=+107.072248745" watchObservedRunningTime="2026-07-14 11:39:33.997333636 +0000 UTC m=+107.073188215" Jul 14 11:39:34.020935 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:34.020309 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" podStartSLOduration=1.596301975 podStartE2EDuration="3.020281963s" totalImagesPullingTime="1.423979988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:34.019595284 +0000 UTC m=+107.095449872" watchObservedRunningTime="2026-07-14 11:39:34.020281963 +0000 UTC m=+107.096136552" Jul 14 11:39:34.981591 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:34.981557 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" Jul 14 11:39:34.987182 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:34.987152 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-8dk7w" Jul 14 11:39:36.990817 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:36.990785 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9qsj" event={"ID":"123b2e42-1ec7-459b-ae70-872058d5465e","Type":"ContainerStarted","Data":"db2e9834d39d6b614152d5459f800fc29d2a9cd4bcbf955771f13fd0e54aabd0"} Jul 14 11:39:36.992283 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:36.992260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfb7987c8-qh4pl" event={"ID":"bc4e54fb-a1a6-43ec-865c-9994a9ac929e","Type":"ContainerStarted","Data":"6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514"} Jul 14 11:39:37.013340 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:37.013300 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w9qsj" podStartSLOduration=32.941002976 podStartE2EDuration="41.01328787s" totalImagesPullingTime="8.072284894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 11:38:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:37.011256553 +0000 UTC m=+110.087111141" watchObservedRunningTime="2026-07-14 11:39:37.01328787 +0000 UTC m=+110.089142457" Jul 14 11:39:37.030973 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:37.030935 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bfb7987c8-qh4pl" podStartSLOduration=1.903662689 podStartE2EDuration="6.030922012s" totalImagesPullingTime="4.127259323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:37.029786436 +0000 UTC m=+110.105641035" watchObservedRunningTime="2026-07-14 11:39:37.030922012 +0000 UTC m=+110.106776600" Jul 14 11:39:38.952964 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:38.952928 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-246h5" Jul 14 11:39:41.492274 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.492243 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ggjzd"] Jul 14 11:39:41.494396 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.494379 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.498414 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.498159 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 11:39:41.498414 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.498213 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 11:39:41.498614 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.498487 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 11:39:41.499596 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.499576 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 11:39:41.499672 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.499594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 11:39:41.499800 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.499779 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 11:39:41.499800 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.499807 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h29qj\"" Jul 14 11:39:41.640189 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.640162 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-wtmp\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.640309 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.640198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.640309 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.640217 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86306431-e7f5-442f-94df-140f5fbdd1d1-sys\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.640309 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.640238 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdz4\" (UniqueName: \"kubernetes.io/projected/86306431-e7f5-442f-94df-140f5fbdd1d1-kube-api-access-chdz4\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.640309 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.640299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86306431-e7f5-442f-94df-140f5fbdd1d1-root\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.640510 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.640378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-textfile\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.640510 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.640416 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86306431-e7f5-442f-94df-140f5fbdd1d1-metrics-client-ca\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.640510 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.640436 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-accelerators-collector-config\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.640510 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.640452 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-tls\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741487 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741462 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86306431-e7f5-442f-94df-140f5fbdd1d1-sys\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741487 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdz4\" (UniqueName: \"kubernetes.io/projected/86306431-e7f5-442f-94df-140f5fbdd1d1-kube-api-access-chdz4\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86306431-e7f5-442f-94df-140f5fbdd1d1-root\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-textfile\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86306431-e7f5-442f-94df-140f5fbdd1d1-metrics-client-ca\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-accelerators-collector-config\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-tls\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741596 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86306431-e7f5-442f-94df-140f5fbdd1d1-root\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741598 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86306431-e7f5-442f-94df-140f5fbdd1d1-sys\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.741661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-wtmp\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.742038 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.742038 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741832 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-wtmp\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.742038 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.741906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-textfile\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.742308 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.742248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86306431-e7f5-442f-94df-140f5fbdd1d1-metrics-client-ca\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.742409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.742366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-accelerators-collector-config\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.743944 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.743924 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.744127 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.744111 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86306431-e7f5-442f-94df-140f5fbdd1d1-node-exporter-tls\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.751141 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.751120 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdz4\" (UniqueName: \"kubernetes.io/projected/86306431-e7f5-442f-94df-140f5fbdd1d1-kube-api-access-chdz4\") pod \"node-exporter-ggjzd\" (UID: \"86306431-e7f5-442f-94df-140f5fbdd1d1\") " pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.804280 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:41.804258 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ggjzd" Jul 14 11:39:41.814136 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:41.814115 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86306431_e7f5_442f_94df_140f5fbdd1d1.slice/crio-4070497b62d85977861c02bfbbf528e220f80d9097e3e3b0f513bd120a44648e WatchSource:0}: Error finding container 4070497b62d85977861c02bfbbf528e220f80d9097e3e3b0f513bd120a44648e: Status 404 returned error can't find the container with id 4070497b62d85977861c02bfbbf528e220f80d9097e3e3b0f513bd120a44648e Jul 14 11:39:42.012894 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.012827 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ggjzd" event={"ID":"86306431-e7f5-442f-94df-140f5fbdd1d1","Type":"ContainerStarted","Data":"4070497b62d85977861c02bfbbf528e220f80d9097e3e3b0f513bd120a44648e"} Jul 14 11:39:42.036413 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.036389 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:42.036510 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.036437 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:42.040822 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.040802 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:42.503917 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.503884 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 11:39:42.506961 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.506940 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.509799 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.509775 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 11:39:42.509918 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.509774 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 11:39:42.510105 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.510070 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 11:39:42.510196 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.510116 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 11:39:42.510196 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.510129 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 11:39:42.510196 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.510096 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 11:39:42.510196 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.510117 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 11:39:42.510385 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.510240 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rplvt\"" Jul 14 11:39:42.510385 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.510317 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 11:39:42.510702 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.510680 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 11:39:42.528589 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.528567 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 11:39:42.649299 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649469 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649469 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649335 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649469 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-config-volume\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-config-out\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kmx\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-kube-api-access-n7kmx\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649753 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-web-config\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649753 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649699 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649753 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649726 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649915 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.649915 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.649811 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.750340 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750313 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.750340 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.750572 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.750572 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-config-volume\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.750572 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:42.750506 2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 14 11:39:42.750723 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:39:42.750576 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-main-tls podName:e12b0433-0d44-4501-b631-dbde1320ea17 nodeName:}" failed. No retries permitted until 2026-07-14 11:39:43.250556882 +0000 UTC m=+116.326411493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17") : secret "alertmanager-main-tls" not found Jul 14 11:39:42.750723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-config-out\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.750723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kmx\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-kube-api-access-n7kmx\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.750723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.750723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.750723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-web-config\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.751029 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750740 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.751029 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.751029 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.751029 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.750867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.751413 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.751299 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.752240 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.751833 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.752240 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.752008 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.754986 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.754500 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-web-config\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.754986 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.754867 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.755630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.755290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.755630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.755419 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-config-volume\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.755630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.755427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.755630 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.755595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.756039 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.756015 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-config-out\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.756344 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.756320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:42.764561 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:42.764539 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kmx\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-kube-api-access-n7kmx\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:43.017558 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:43.017491 2562 generic.go:356] "Generic (PLEG): container finished" podID="86306431-e7f5-442f-94df-140f5fbdd1d1" containerID="e5b1eac16a7bd072e403a92e4af7c314d949171bb5d7a6f515c5a0937477de7b" exitCode=0 Jul 14 11:39:43.017661 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:43.017584 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ggjzd" event={"ID":"86306431-e7f5-442f-94df-140f5fbdd1d1","Type":"ContainerDied","Data":"e5b1eac16a7bd072e403a92e4af7c314d949171bb5d7a6f515c5a0937477de7b"} Jul 14 11:39:43.021497 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:43.021477 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:39:43.254555 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:43.254528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:43.256671 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:43.256648 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:43.418790 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:43.418735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:39:43.555032 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:43.555009 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 11:39:43.556407 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:43.556381 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12b0433_0d44_4501_b631_dbde1320ea17.slice/crio-ff45efbe0e90c69a9368b1867af81ee6ed01598a58ab8a969c841e9f775481b9 WatchSource:0}: Error finding container ff45efbe0e90c69a9368b1867af81ee6ed01598a58ab8a969c841e9f775481b9: Status 404 returned error can't find the container with id ff45efbe0e90c69a9368b1867af81ee6ed01598a58ab8a969c841e9f775481b9 Jul 14 11:39:44.023204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:44.023175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ggjzd" event={"ID":"86306431-e7f5-442f-94df-140f5fbdd1d1","Type":"ContainerStarted","Data":"5213c9bccba5a5bb33f9d46992360cd2de85f2f0cde97db8f4e2925d9721d73f"} Jul 14 11:39:44.023336 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:44.023229 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ggjzd" event={"ID":"86306431-e7f5-442f-94df-140f5fbdd1d1","Type":"ContainerStarted","Data":"9258d00a1741edf760ecd8fc2259112297171a171833c508efc3e48b5f080e1e"} Jul 14 11:39:44.028390 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:44.028366 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerStarted","Data":"ff45efbe0e90c69a9368b1867af81ee6ed01598a58ab8a969c841e9f775481b9"} Jul 14 11:39:44.046874 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:44.046824 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ggjzd" podStartSLOduration=2.001345317 podStartE2EDuration="3.046811972s" totalImagesPullingTime="982.876821ms" totalInitContainerRuntime="62.589834ms" isStatefulPod=true podCreationTimestamp="2026-07-14 11:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:44.045388889 +0000 UTC m=+117.121243479" watchObservedRunningTime="2026-07-14 11:39:44.046811972 +0000 UTC m=+117.122666560" Jul 14 11:39:45.033436 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:45.033402 2562 generic.go:356] "Generic (PLEG): container finished" podID="e12b0433-0d44-4501-b631-dbde1320ea17" containerID="e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870" exitCode=0 Jul 14 11:39:45.033788 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:45.033491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerDied","Data":"e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870"} Jul 14 11:39:47.044328 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:47.044244 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerStarted","Data":"2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5"} Jul 14 11:39:47.044328 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:47.044307 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerStarted","Data":"f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a"} Jul 14 11:39:47.044690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:47.044332 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerStarted","Data":"6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248"} Jul 14 11:39:47.044690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:47.044355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerStarted","Data":"6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f"} Jul 14 11:39:47.044690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:47.044377 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerStarted","Data":"17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741"} Jul 14 11:39:48.051442 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:48.051408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerStarted","Data":"13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2"} Jul 14 11:39:48.088818 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:48.088766 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.8576211540000003 podStartE2EDuration="6.088750502s" totalImagesPullingTime="3.214071227s" totalInitContainerRuntime="17.058121ms" isStatefulPod=false podCreationTimestamp="2026-07-14 11:39:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:48.087348346 +0000 UTC m=+121.163202946" watchObservedRunningTime="2026-07-14 11:39:48.088750502 +0000 UTC m=+121.164605089" Jul 14 11:39:53.982486 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:53.982450 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c46ccf68d-5rwrx" Jul 14 11:39:57.260180 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:57.260135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:39:57.262725 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:57.262699 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb5ccae-f66d-42fe-bfca-8016ee9904f8-metrics-certs\") pod \"network-metrics-daemon-t2c79\" (UID: \"eeb5ccae-f66d-42fe-bfca-8016ee9904f8\") " pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:39:57.299450 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:57.299421 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-26j7k\"" Jul 14 11:39:57.306808 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:57.306791 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t2c79" Jul 14 11:39:57.433115 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:57.433050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t2c79"] Jul 14 11:39:57.435049 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:39:57.435024 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb5ccae_f66d_42fe_bfca_8016ee9904f8.slice/crio-f7188a580eddd741e55c935926ad3441861c577d4dce7a3ed7eb0c97f0c20dbc WatchSource:0}: Error finding container f7188a580eddd741e55c935926ad3441861c577d4dce7a3ed7eb0c97f0c20dbc: Status 404 returned error can't find the container with id f7188a580eddd741e55c935926ad3441861c577d4dce7a3ed7eb0c97f0c20dbc Jul 14 11:39:58.092981 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:58.092945 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t2c79" event={"ID":"eeb5ccae-f66d-42fe-bfca-8016ee9904f8","Type":"ContainerStarted","Data":"f7188a580eddd741e55c935926ad3441861c577d4dce7a3ed7eb0c97f0c20dbc"} Jul 14 11:39:59.097864 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:59.097826 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t2c79" event={"ID":"eeb5ccae-f66d-42fe-bfca-8016ee9904f8","Type":"ContainerStarted","Data":"3c7036c0037505ed7e81d8bbb1c83f4593f432dfc13c0ae91ca7b84f640c7e8c"} Jul 14 11:39:59.098340 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:59.097885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t2c79" event={"ID":"eeb5ccae-f66d-42fe-bfca-8016ee9904f8","Type":"ContainerStarted","Data":"359e6e35e547e894e1527284e7e48c85d164b2e90104e2e4693d8a5cdc20e8ee"} Jul 14 11:39:59.118850 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:39:59.118800 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t2c79" podStartSLOduration=130.210054094 podStartE2EDuration="2m11.118784417s" totalImagesPullingTime="908.730323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:39:59.118067367 +0000 UTC m=+132.193921955" watchObservedRunningTime="2026-07-14 11:39:59.118784417 +0000 UTC m=+132.194639005" Jul 14 11:40:02.057116 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.057073 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fdf958b99-82p5g"] Jul 14 11:40:02.059288 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.059270 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.088716 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.088693 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fdf958b99-82p5g"] Jul 14 11:40:02.091619 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.091594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-oauth-config\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.091723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.091634 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-serving-cert\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.091723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.091665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-trusted-ca-bundle\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.091723 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.091691 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr4bj\" (UniqueName: \"kubernetes.io/projected/771b1e14-9301-49a6-a09a-ce19ff1871b9-kube-api-access-wr4bj\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.091883 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.091765 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-service-ca\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.091883 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.091794 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-oauth-serving-cert\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.091883 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.091820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-config\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.192990 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.192965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-config\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.193121 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.192995 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-oauth-config\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.193121 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.193019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-serving-cert\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.193121 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.193046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-trusted-ca-bundle\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.193121 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.193067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr4bj\" (UniqueName: \"kubernetes.io/projected/771b1e14-9301-49a6-a09a-ce19ff1871b9-kube-api-access-wr4bj\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.193287 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.193249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-service-ca\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.193344 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.193297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-oauth-serving-cert\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.193758 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.193724 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-config\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.193939 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.193914 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-trusted-ca-bundle\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.194006 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.193935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-oauth-serving-cert\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.194288 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.194265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-service-ca\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.195496 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.195476 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-serving-cert\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.195613 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.195593 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-oauth-config\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.231667 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.231648 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr4bj\" (UniqueName: \"kubernetes.io/projected/771b1e14-9301-49a6-a09a-ce19ff1871b9-kube-api-access-wr4bj\") pod \"console-6fdf958b99-82p5g\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.368435 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.368390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:02.497928 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:02.497906 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fdf958b99-82p5g"] Jul 14 11:40:02.500129 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:40:02.500105 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771b1e14_9301_49a6_a09a_ce19ff1871b9.slice/crio-ebc24ce7db01970d6704d316b06a95638c1c31edf326e0d785b18b6c2505f6d5 WatchSource:0}: Error finding container ebc24ce7db01970d6704d316b06a95638c1c31edf326e0d785b18b6c2505f6d5: Status 404 returned error can't find the container with id ebc24ce7db01970d6704d316b06a95638c1c31edf326e0d785b18b6c2505f6d5 Jul 14 11:40:03.113211 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:03.113174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdf958b99-82p5g" event={"ID":"771b1e14-9301-49a6-a09a-ce19ff1871b9","Type":"ContainerStarted","Data":"a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87"} Jul 14 11:40:03.113563 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:03.113223 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdf958b99-82p5g" event={"ID":"771b1e14-9301-49a6-a09a-ce19ff1871b9","Type":"ContainerStarted","Data":"ebc24ce7db01970d6704d316b06a95638c1c31edf326e0d785b18b6c2505f6d5"} Jul 14 11:40:03.137905 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:03.137851 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fdf958b99-82p5g" podStartSLOduration=1.137837211 podStartE2EDuration="1.137837211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:40:03.136797054 +0000 UTC m=+136.212651642" watchObservedRunningTime="2026-07-14 11:40:03.137837211 +0000 UTC m=+136.213691797" Jul 14 11:40:12.147182 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:12.147151 2562 generic.go:356] "Generic (PLEG): container finished" podID="d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0" containerID="2a42061192c92ef6fbfc2976b0c0635fa42b02d64da88f26d104cafbbde94559" exitCode=0 Jul 14 11:40:12.147586 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:12.147232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" event={"ID":"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0","Type":"ContainerDied","Data":"2a42061192c92ef6fbfc2976b0c0635fa42b02d64da88f26d104cafbbde94559"} Jul 14 11:40:12.147586 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:12.147510 2562 scope.go:117] "RemoveContainer" containerID="2a42061192c92ef6fbfc2976b0c0635fa42b02d64da88f26d104cafbbde94559" Jul 14 11:40:12.369158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:12.369132 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:12.369319 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:12.369215 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:12.373762 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:12.373740 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:13.153137 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:13.153104 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-wnpqp" event={"ID":"d4bc14e0-adc9-429f-a2c6-2f6c1516b8b0","Type":"ContainerStarted","Data":"a8094e75ae7db68ae82e60aae7d2e5ca09d50199e4d9f716861ef0bf9f932292"} Jul 14 11:40:13.156828 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:13.156804 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:40:13.223655 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:13.223625 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bfb7987c8-qh4pl"] Jul 14 11:40:38.251847 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.251794 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bfb7987c8-qh4pl" podUID="bc4e54fb-a1a6-43ec-865c-9994a9ac929e" containerName="console" containerID="cri-o://6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514" gracePeriod=15 Jul 14 11:40:38.532315 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.532282 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:40:38.658023 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.657979 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-oauth-config\") pod \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " Jul 14 11:40:38.658238 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.658043 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-trusted-ca-bundle\") pod \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " Jul 14 11:40:38.658238 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.658098 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn9mt\" (UniqueName: \"kubernetes.io/projected/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-kube-api-access-rn9mt\") pod \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " Jul 14 11:40:38.658238 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.658132 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-oauth-serving-cert\") pod \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " Jul 14 11:40:38.658238 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.658174 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-config\") pod \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " Jul 14 11:40:38.658462 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.658275 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-service-ca\") pod \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " Jul 14 11:40:38.658462 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.658309 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-serving-cert\") pod \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\" (UID: \"bc4e54fb-a1a6-43ec-865c-9994a9ac929e\") " Jul 14 11:40:38.659787 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.659214 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-config" (OuterVolumeSpecName: "console-config") pod "bc4e54fb-a1a6-43ec-865c-9994a9ac929e" (UID: "bc4e54fb-a1a6-43ec-865c-9994a9ac929e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:40:38.659787 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.659435 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bc4e54fb-a1a6-43ec-865c-9994a9ac929e" (UID: "bc4e54fb-a1a6-43ec-865c-9994a9ac929e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:40:38.659787 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.659494 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-service-ca" (OuterVolumeSpecName: "service-ca") pod "bc4e54fb-a1a6-43ec-865c-9994a9ac929e" (UID: "bc4e54fb-a1a6-43ec-865c-9994a9ac929e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:40:38.659787 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.659756 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bc4e54fb-a1a6-43ec-865c-9994a9ac929e" (UID: "bc4e54fb-a1a6-43ec-865c-9994a9ac929e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:40:38.661922 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.661876 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bc4e54fb-a1a6-43ec-865c-9994a9ac929e" (UID: "bc4e54fb-a1a6-43ec-865c-9994a9ac929e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:40:38.663241 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.663167 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bc4e54fb-a1a6-43ec-865c-9994a9ac929e" (UID: "bc4e54fb-a1a6-43ec-865c-9994a9ac929e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:40:38.663730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.663690 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-kube-api-access-rn9mt" (OuterVolumeSpecName: "kube-api-access-rn9mt") pod "bc4e54fb-a1a6-43ec-865c-9994a9ac929e" (UID: "bc4e54fb-a1a6-43ec-865c-9994a9ac929e"). InnerVolumeSpecName "kube-api-access-rn9mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:40:38.759757 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.759717 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:40:38.759757 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.759749 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-serving-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:40:38.759757 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.759759 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-oauth-config\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:40:38.759757 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.759767 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-trusted-ca-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:40:38.760024 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.759799 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn9mt\" (UniqueName: \"kubernetes.io/projected/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-kube-api-access-rn9mt\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:40:38.760024 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.759809 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-oauth-serving-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:40:38.760024 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:38.759817 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4e54fb-a1a6-43ec-865c-9994a9ac929e-console-config\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:40:39.254256 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.254222 2562 generic.go:356] "Generic (PLEG): container finished" podID="bc4e54fb-a1a6-43ec-865c-9994a9ac929e" containerID="6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514" exitCode=0 Jul 14 11:40:39.254668 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.254268 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfb7987c8-qh4pl" Jul 14 11:40:39.254668 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.254312 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfb7987c8-qh4pl" event={"ID":"bc4e54fb-a1a6-43ec-865c-9994a9ac929e","Type":"ContainerDied","Data":"6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514"} Jul 14 11:40:39.254668 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.254414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfb7987c8-qh4pl" event={"ID":"bc4e54fb-a1a6-43ec-865c-9994a9ac929e","Type":"ContainerDied","Data":"7e105ce6620eab03c83afa6d5c9fe32d4a33a7b2f941c8abb5922e79bc1262f3"} Jul 14 11:40:39.254668 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.254447 2562 scope.go:117] "RemoveContainer" containerID="6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514" Jul 14 11:40:39.262425 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.262412 2562 scope.go:117] "RemoveContainer" containerID="6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514" Jul 14 11:40:39.262646 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:40:39.262625 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514\": container with ID starting with 6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514 not found: ID does not exist" containerID="6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514" Jul 14 11:40:39.262697 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.262653 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514"} err="failed to get container status \"6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514\": rpc error: code = NotFound desc = could not find container \"6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514\": container with ID starting with 6228dce8fcd28508d5aa74d60de536f2697ffeaa0b9a8e7f0e88b58410477514 not found: ID does not exist" Jul 14 11:40:39.276732 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.276706 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bfb7987c8-qh4pl"] Jul 14 11:40:39.286672 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.286649 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bfb7987c8-qh4pl"] Jul 14 11:40:39.478486 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:40:39.478456 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc4e54fb-a1a6-43ec-865c-9994a9ac929e" path="/var/lib/kubelet/pods/bc4e54fb-a1a6-43ec-865c-9994a9ac929e/volumes" Jul 14 11:41:01.857829 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:01.857797 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 11:41:01.858371 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:01.858071 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="alertmanager" containerID="cri-o://17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741" gracePeriod=120 Jul 14 11:41:01.858371 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:01.858127 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy-web" containerID="cri-o://6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248" gracePeriod=120 Jul 14 11:41:01.858371 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:01.858136 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy-metric" containerID="cri-o://2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5" gracePeriod=120 Jul 14 11:41:01.858371 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:01.858192 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="prom-label-proxy" containerID="cri-o://13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2" gracePeriod=120 Jul 14 11:41:01.858371 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:01.858223 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="config-reloader" containerID="cri-o://6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f" gracePeriod=120 Jul 14 11:41:01.858371 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:01.858201 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy" containerID="cri-o://f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a" gracePeriod=120 Jul 14 11:41:02.342101 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:02.342054 2562 generic.go:356] "Generic (PLEG): container finished" podID="e12b0433-0d44-4501-b631-dbde1320ea17" containerID="13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2" exitCode=0 Jul 14 11:41:02.342101 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:02.342093 2562 generic.go:356] "Generic (PLEG): container finished" podID="e12b0433-0d44-4501-b631-dbde1320ea17" containerID="f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a" exitCode=0 Jul 14 11:41:02.342101 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:02.342100 2562 generic.go:356] "Generic (PLEG): container finished" podID="e12b0433-0d44-4501-b631-dbde1320ea17" containerID="6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f" exitCode=0 Jul 14 11:41:02.342101 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:02.342109 2562 generic.go:356] "Generic (PLEG): container finished" podID="e12b0433-0d44-4501-b631-dbde1320ea17" containerID="17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741" exitCode=0 Jul 14 11:41:02.342374 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:02.342128 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerDied","Data":"13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2"} Jul 14 11:41:02.342374 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:02.342194 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerDied","Data":"f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a"} Jul 14 11:41:02.342374 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:02.342220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerDied","Data":"6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f"} Jul 14 11:41:02.342374 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:02.342244 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerDied","Data":"17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741"} Jul 14 11:41:03.098777 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.098755 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.252132 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252103 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-tls-assets\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252304 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252145 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252304 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252175 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-web\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252304 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252194 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-metrics-client-ca\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252466 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252318 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-config-out\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252466 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252370 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-main-tls\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252466 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252396 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-web-config\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252466 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252422 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-trusted-ca-bundle\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252466 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252457 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-cluster-tls-config\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252699 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252494 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-main-db\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252699 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252523 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7kmx\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-kube-api-access-n7kmx\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252699 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252553 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252699 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252553 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:41:03.252699 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252598 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-config-volume\") pod \"e12b0433-0d44-4501-b631-dbde1320ea17\" (UID: \"e12b0433-0d44-4501-b631-dbde1320ea17\") " Jul 14 11:41:03.252921 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.252855 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-metrics-client-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.253029 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.253001 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:41:03.254962 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.254932 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:41:03.255284 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.254964 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-config-out" (OuterVolumeSpecName: "config-out") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:41:03.255396 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.255367 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:41:03.255455 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.255374 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:41:03.255671 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.255622 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:41:03.255671 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.255625 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-config-volume" (OuterVolumeSpecName: "config-volume") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:41:03.255832 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.255793 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:41:03.257132 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.257067 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-kube-api-access-n7kmx" (OuterVolumeSpecName: "kube-api-access-n7kmx") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "kube-api-access-n7kmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:41:03.257132 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.257102 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:41:03.259095 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.259062 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:41:03.265443 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.265424 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-web-config" (OuterVolumeSpecName: "web-config") pod "e12b0433-0d44-4501-b631-dbde1320ea17" (UID: "e12b0433-0d44-4501-b631-dbde1320ea17"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:41:03.349731 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.349699 2562 generic.go:356] "Generic (PLEG): container finished" podID="e12b0433-0d44-4501-b631-dbde1320ea17" containerID="2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5" exitCode=0 Jul 14 11:41:03.349731 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.349724 2562 generic.go:356] "Generic (PLEG): container finished" podID="e12b0433-0d44-4501-b631-dbde1320ea17" containerID="6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248" exitCode=0 Jul 14 11:41:03.349980 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.349775 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.349980 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.349784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerDied","Data":"2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5"} Jul 14 11:41:03.349980 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.349875 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerDied","Data":"6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248"} Jul 14 11:41:03.349980 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.349914 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e12b0433-0d44-4501-b631-dbde1320ea17","Type":"ContainerDied","Data":"ff45efbe0e90c69a9368b1867af81ee6ed01598a58ab8a969c841e9f775481b9"} Jul 14 11:41:03.349980 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.349937 2562 scope.go:117] "RemoveContainer" containerID="13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2" Jul 14 11:41:03.353265 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353246 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-config-out\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353265 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353264 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-main-tls\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353273 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-web-config\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353284 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353294 2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-cluster-tls-config\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353302 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e12b0433-0d44-4501-b631-dbde1320ea17-alertmanager-main-db\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353310 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7kmx\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-kube-api-access-n7kmx\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353318 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353327 2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-config-volume\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353335 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e12b0433-0d44-4501-b631-dbde1320ea17-tls-assets\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353342 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.353409 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.353350 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e12b0433-0d44-4501-b631-dbde1320ea17-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:03.357576 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.357550 2562 scope.go:117] "RemoveContainer" containerID="2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5" Jul 14 11:41:03.364334 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.364313 2562 scope.go:117] "RemoveContainer" containerID="f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a" Jul 14 11:41:03.371087 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.371061 2562 scope.go:117] "RemoveContainer" containerID="6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248" Jul 14 11:41:03.377431 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.377412 2562 scope.go:117] "RemoveContainer" containerID="6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f" Jul 14 11:41:03.380250 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.380227 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 11:41:03.384614 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.384593 2562 scope.go:117] "RemoveContainer" containerID="17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741" Jul 14 11:41:03.386514 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.386496 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 11:41:03.391377 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.391360 2562 scope.go:117] "RemoveContainer" containerID="e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870" Jul 14 11:41:03.397782 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.397759 2562 scope.go:117] "RemoveContainer" containerID="13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2" Jul 14 11:41:03.398036 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:41:03.398011 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2\": container with ID starting with 13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2 not found: ID does not exist" containerID="13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2" Jul 14 11:41:03.398111 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.398044 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2"} err="failed to get container status \"13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2\": rpc error: code = NotFound desc = could not find container \"13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2\": container with ID starting with 13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2 not found: ID does not exist" Jul 14 11:41:03.398111 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.398063 2562 scope.go:117] "RemoveContainer" containerID="2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5" Jul 14 11:41:03.398321 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:41:03.398308 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5\": container with ID starting with 2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5 not found: ID does not exist" containerID="2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5" Jul 14 11:41:03.398359 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.398324 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5"} err="failed to get container status \"2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5\": rpc error: code = NotFound desc = could not find container \"2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5\": container with ID starting with 2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5 not found: ID does not exist" Jul 14 11:41:03.398359 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.398342 2562 scope.go:117] "RemoveContainer" containerID="f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a" Jul 14 11:41:03.398583 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:41:03.398562 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a\": container with ID starting with f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a not found: ID does not exist" containerID="f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a" Jul 14 11:41:03.398621 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.398589 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a"} err="failed to get container status \"f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a\": rpc error: code = NotFound desc = could not find container \"f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a\": container with ID starting with f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a not found: ID does not exist" Jul 14 11:41:03.398621 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.398605 2562 scope.go:117] "RemoveContainer" containerID="6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248" Jul 14 11:41:03.398844 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:41:03.398830 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248\": container with ID starting with 6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248 not found: ID does not exist" containerID="6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248" Jul 14 11:41:03.398882 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.398848 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248"} err="failed to get container status \"6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248\": rpc error: code = NotFound desc = could not find container \"6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248\": container with ID starting with 6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248 not found: ID does not exist" Jul 14 11:41:03.398882 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.398860 2562 scope.go:117] "RemoveContainer" containerID="6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f" Jul 14 11:41:03.399070 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:41:03.399052 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f\": container with ID starting with 6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f not found: ID does not exist" containerID="6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f" Jul 14 11:41:03.399137 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399072 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f"} err="failed to get container status \"6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f\": rpc error: code = NotFound desc = could not find container \"6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f\": container with ID starting with 6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f not found: ID does not exist" Jul 14 11:41:03.399137 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399105 2562 scope.go:117] "RemoveContainer" containerID="17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741" Jul 14 11:41:03.399314 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:41:03.399299 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741\": container with ID starting with 17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741 not found: ID does not exist" containerID="17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741" Jul 14 11:41:03.399351 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399320 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741"} err="failed to get container status \"17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741\": rpc error: code = NotFound desc = could not find container \"17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741\": container with ID starting with 17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741 not found: ID does not exist" Jul 14 11:41:03.399351 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399332 2562 scope.go:117] "RemoveContainer" containerID="e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870" Jul 14 11:41:03.399528 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:41:03.399513 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870\": container with ID starting with e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870 not found: ID does not exist" containerID="e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870" Jul 14 11:41:03.399563 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399531 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870"} err="failed to get container status \"e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870\": rpc error: code = NotFound desc = could not find container \"e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870\": container with ID starting with e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870 not found: ID does not exist" Jul 14 11:41:03.399563 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399542 2562 scope.go:117] "RemoveContainer" containerID="13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2" Jul 14 11:41:03.399728 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399714 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2"} err="failed to get container status \"13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2\": rpc error: code = NotFound desc = could not find container \"13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2\": container with ID starting with 13d2ae1eb9c091f240cbc4e2e3176a46b9c29a1543bbbd6ac1d3646eb13bb4a2 not found: ID does not exist" Jul 14 11:41:03.399763 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399727 2562 scope.go:117] "RemoveContainer" containerID="2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5" Jul 14 11:41:03.399924 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399909 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5"} err="failed to get container status \"2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5\": rpc error: code = NotFound desc = could not find container \"2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5\": container with ID starting with 2eb2d0693a69bfcbf0fafea470cedfbc51042dc284622d12bfbd743927ca0ee5 not found: ID does not exist" Jul 14 11:41:03.399984 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.399924 2562 scope.go:117] "RemoveContainer" containerID="f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a" Jul 14 11:41:03.400234 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.400219 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a"} err="failed to get container status \"f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a\": rpc error: code = NotFound desc = could not find container \"f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a\": container with ID starting with f7da3a4d940a7a9fdb3c41be099f8d1c428b7c6c63c7eb0891b1f3d8e1723e4a not found: ID does not exist" Jul 14 11:41:03.400234 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.400233 2562 scope.go:117] "RemoveContainer" containerID="6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248" Jul 14 11:41:03.400412 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.400399 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248"} err="failed to get container status \"6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248\": rpc error: code = NotFound desc = could not find container \"6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248\": container with ID starting with 6b0e9107d02fd6a94189729efa39d4bbec0f36ab970c4a487662b8289b201248 not found: ID does not exist" Jul 14 11:41:03.400412 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.400412 2562 scope.go:117] "RemoveContainer" containerID="6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f" Jul 14 11:41:03.400570 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.400556 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f"} err="failed to get container status \"6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f\": rpc error: code = NotFound desc = could not find container \"6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f\": container with ID starting with 6f39c728c82b1ce5ad86975b4e03605af14c50ac9d502bb8ed9a6e6dfd6ac79f not found: ID does not exist" Jul 14 11:41:03.400608 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.400570 2562 scope.go:117] "RemoveContainer" containerID="17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741" Jul 14 11:41:03.400767 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.400747 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741"} err="failed to get container status \"17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741\": rpc error: code = NotFound desc = could not find container \"17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741\": container with ID starting with 17a1b419370a601dc1e2608b1849f2dbc16753650bc55374a942b5236540a741 not found: ID does not exist" Jul 14 11:41:03.400806 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.400769 2562 scope.go:117] "RemoveContainer" containerID="e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870" Jul 14 11:41:03.400969 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.400953 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870"} err="failed to get container status \"e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870\": rpc error: code = NotFound desc = could not find container \"e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870\": container with ID starting with e124e35edcc27d548d3107ff701c5b9cbfe38e6decadccb51190bfb945621870 not found: ID does not exist" Jul 14 11:41:03.419946 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.419924 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 11:41:03.420199 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420185 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="alertmanager" Jul 14 11:41:03.420243 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420203 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="alertmanager" Jul 14 11:41:03.420243 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420219 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="prom-label-proxy" Jul 14 11:41:03.420243 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420225 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="prom-label-proxy" Jul 14 11:41:03.420243 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420234 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy" Jul 14 11:41:03.420243 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420240 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420247 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4e54fb-a1a6-43ec-865c-9994a9ac929e" containerName="console" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420253 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4e54fb-a1a6-43ec-865c-9994a9ac929e" containerName="console" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420264 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="init-config-reloader" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420269 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="init-config-reloader" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420277 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="config-reloader" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420282 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="config-reloader" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420294 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy-web" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420298 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy-web" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420303 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy-metric" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420308 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy-metric" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420350 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy-web" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420357 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="prom-label-proxy" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420362 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy-metric" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420368 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc4e54fb-a1a6-43ec-865c-9994a9ac929e" containerName="console" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420373 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="config-reloader" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420379 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="alertmanager" Jul 14 11:41:03.420384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.420384 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" containerName="kube-rbac-proxy" Jul 14 11:41:03.424278 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.424263 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.427459 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.427443 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 11:41:03.427560 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.427540 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 11:41:03.427621 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.427578 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 11:41:03.427621 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.427606 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 11:41:03.427725 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.427610 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 11:41:03.427725 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.427542 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 11:41:03.427725 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.427622 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 11:41:03.427725 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.427624 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rplvt\"" Jul 14 11:41:03.427981 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.427907 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 11:41:03.432092 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.432064 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 11:41:03.445264 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.445236 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 11:41:03.478308 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.478286 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e12b0433-0d44-4501-b631-dbde1320ea17" path="/var/lib/kubelet/pods/e12b0433-0d44-4501-b631-dbde1320ea17/volumes" Jul 14 11:41:03.554801 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.554753 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.554801 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.554788 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x749v\" (UniqueName: \"kubernetes.io/projected/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-kube-api-access-x749v\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.554908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.554808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.554908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.554828 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.554908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.554862 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.554908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.554903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.555034 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.554922 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.555034 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.554948 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.555034 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.554973 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-web-config\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.555034 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.555001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.555034 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.555018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.555199 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.555039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-config-out\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.555199 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.555069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.655766 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x749v\" (UniqueName: \"kubernetes.io/projected/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-kube-api-access-x749v\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.655833 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.655833 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.655833 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.655833 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656006 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656006 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655884 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656006 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-web-config\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656006 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656006 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656006 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.655990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-config-out\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656295 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.656012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656295 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.656041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656405 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.656379 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.656788 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.656715 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.657535 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.657210 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.658826 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.658802 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.658967 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.658939 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.659064 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.659023 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.659064 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.659043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.659207 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.659133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-web-config\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.659207 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.659184 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.659400 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.659378 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.660016 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.659998 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.660116 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.660102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-config-out\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.665696 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.665680 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x749v\" (UniqueName: \"kubernetes.io/projected/7ee3ab75-ad74-42f7-a4a0-53e4d126773d-kube-api-access-x749v\") pod \"alertmanager-main-0\" (UID: \"7ee3ab75-ad74-42f7-a4a0-53e4d126773d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.733192 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.733172 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 11:41:03.860828 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:03.860799 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 11:41:03.863583 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:41:03.863555 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee3ab75_ad74_42f7_a4a0_53e4d126773d.slice/crio-1c4ddf234a9c1fe1c7a2454191b9b7570b3f3efbc761f21b3d5a341a177b9632 WatchSource:0}: Error finding container 1c4ddf234a9c1fe1c7a2454191b9b7570b3f3efbc761f21b3d5a341a177b9632: Status 404 returned error can't find the container with id 1c4ddf234a9c1fe1c7a2454191b9b7570b3f3efbc761f21b3d5a341a177b9632 Jul 14 11:41:04.355760 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:04.355731 2562 generic.go:356] "Generic (PLEG): container finished" podID="7ee3ab75-ad74-42f7-a4a0-53e4d126773d" containerID="dc117e2d3b9b1c3b6ca6074bb7eb9fb3ae06907ef8e88cd4853e3a282c654c5c" exitCode=0 Jul 14 11:41:04.356065 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:04.355769 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee3ab75-ad74-42f7-a4a0-53e4d126773d","Type":"ContainerDied","Data":"dc117e2d3b9b1c3b6ca6074bb7eb9fb3ae06907ef8e88cd4853e3a282c654c5c"} Jul 14 11:41:04.356065 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:04.355818 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee3ab75-ad74-42f7-a4a0-53e4d126773d","Type":"ContainerStarted","Data":"1c4ddf234a9c1fe1c7a2454191b9b7570b3f3efbc761f21b3d5a341a177b9632"} Jul 14 11:41:05.362187 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:05.362147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee3ab75-ad74-42f7-a4a0-53e4d126773d","Type":"ContainerStarted","Data":"d42ebc72d258922ccfde956e69c8b1f613da25651d3436d71aba770d520f7177"} Jul 14 11:41:05.362531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:05.362222 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee3ab75-ad74-42f7-a4a0-53e4d126773d","Type":"ContainerStarted","Data":"01677c2e73e9c9321a20af4f6b4e1c92df075b3893ec41a7d1713a0b75dc3c0d"} Jul 14 11:41:05.362531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:05.362250 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee3ab75-ad74-42f7-a4a0-53e4d126773d","Type":"ContainerStarted","Data":"aa85a69c4ef77bd100db2d286f9e879ea13865682c626f9014034b74cf6e00c2"} Jul 14 11:41:05.362531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:05.362276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee3ab75-ad74-42f7-a4a0-53e4d126773d","Type":"ContainerStarted","Data":"ba722acc5399367302136476c0b7a72970f4e44101ce77f25f8cf2cc96411082"} Jul 14 11:41:05.362531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:05.362306 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee3ab75-ad74-42f7-a4a0-53e4d126773d","Type":"ContainerStarted","Data":"c4c8271ce1411ce41828466c6273698c198ba5caa3f1b0a0fff37445ad3ea33a"} Jul 14 11:41:05.362531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:05.362328 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee3ab75-ad74-42f7-a4a0-53e4d126773d","Type":"ContainerStarted","Data":"16e8e38fca5bf00a9dab433484fd0b1a0466e98935f0acd1ef42de9f117982aa"} Jul 14 11:41:05.395528 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:05.395476 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.385312951 podStartE2EDuration="2.395460207s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.147256ms" isStatefulPod=false podCreationTimestamp="2026-07-14 11:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:41:05.393453756 +0000 UTC m=+198.469308356" watchObservedRunningTime="2026-07-14 11:41:05.395460207 +0000 UTC m=+198.471314800" Jul 14 11:41:13.936091 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:13.936037 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b767f6bc-6ngmm"] Jul 14 11:41:13.938503 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:13.938483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:13.951310 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:13.951284 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b767f6bc-6ngmm"] Jul 14 11:41:14.032794 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.032763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-config\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.032794 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.032794 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtqsr\" (UniqueName: \"kubernetes.io/projected/96c1232b-bdcd-4a4b-949b-7b22205f648f-kube-api-access-gtqsr\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.032982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.032817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-serving-cert\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.032982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.032844 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-service-ca\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.032982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.032921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-oauth-config\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.032982 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.032959 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-oauth-serving-cert\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.033173 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.033007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-trusted-ca-bundle\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.134164 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.134126 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-config\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.134164 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.134164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtqsr\" (UniqueName: \"kubernetes.io/projected/96c1232b-bdcd-4a4b-949b-7b22205f648f-kube-api-access-gtqsr\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.134369 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.134181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-serving-cert\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.134369 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.134210 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-service-ca\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.134369 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.134240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-oauth-config\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.134369 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.134273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-oauth-serving-cert\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.134369 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.134309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-trusted-ca-bundle\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.134943 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.134918 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-service-ca\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.135062 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.134920 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-config\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.135062 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.135012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-oauth-serving-cert\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.135309 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.135285 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-trusted-ca-bundle\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.136695 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.136676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-serving-cert\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.136796 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.136706 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-oauth-config\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.145035 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.145012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtqsr\" (UniqueName: \"kubernetes.io/projected/96c1232b-bdcd-4a4b-949b-7b22205f648f-kube-api-access-gtqsr\") pod \"console-84b767f6bc-6ngmm\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.249515 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.249482 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:14.387393 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.387367 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b767f6bc-6ngmm"] Jul 14 11:41:14.389142 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:41:14.389113 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c1232b_bdcd_4a4b_949b_7b22205f648f.slice/crio-7b3caca037e02d59909278702f52becafca2c333e6c8b9e49b7c8d8abd7b4f4d WatchSource:0}: Error finding container 7b3caca037e02d59909278702f52becafca2c333e6c8b9e49b7c8d8abd7b4f4d: Status 404 returned error can't find the container with id 7b3caca037e02d59909278702f52becafca2c333e6c8b9e49b7c8d8abd7b4f4d Jul 14 11:41:14.401629 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:14.401602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b767f6bc-6ngmm" event={"ID":"96c1232b-bdcd-4a4b-949b-7b22205f648f","Type":"ContainerStarted","Data":"7b3caca037e02d59909278702f52becafca2c333e6c8b9e49b7c8d8abd7b4f4d"} Jul 14 11:41:15.407262 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:15.407232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b767f6bc-6ngmm" event={"ID":"96c1232b-bdcd-4a4b-949b-7b22205f648f","Type":"ContainerStarted","Data":"2534a288cf494934cce7c5cc89fc31e5c6d1db2b74f112ebb5caf66850faf400"} Jul 14 11:41:15.430018 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:15.429974 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84b767f6bc-6ngmm" podStartSLOduration=2.429959006 podStartE2EDuration="2.429959006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:41:15.429489109 +0000 UTC m=+208.505343698" watchObservedRunningTime="2026-07-14 11:41:15.429959006 +0000 UTC m=+208.505813592" Jul 14 11:41:24.249676 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:24.249642 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:24.250141 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:24.249703 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:24.253898 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:24.253869 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:24.442223 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:24.442202 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:41:24.499069 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:24.499037 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fdf958b99-82p5g"] Jul 14 11:41:49.520946 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.520893 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fdf958b99-82p5g" podUID="771b1e14-9301-49a6-a09a-ce19ff1871b9" containerName="console" containerID="cri-o://a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87" gracePeriod=15 Jul 14 11:41:49.760592 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.760571 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:41:49.872625 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.872565 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-config\") pod \"771b1e14-9301-49a6-a09a-ce19ff1871b9\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " Jul 14 11:41:49.872625 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.872600 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-serving-cert\") pod \"771b1e14-9301-49a6-a09a-ce19ff1871b9\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " Jul 14 11:41:49.872625 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.872621 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-oauth-config\") pod \"771b1e14-9301-49a6-a09a-ce19ff1871b9\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " Jul 14 11:41:49.872841 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.872639 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-trusted-ca-bundle\") pod \"771b1e14-9301-49a6-a09a-ce19ff1871b9\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " Jul 14 11:41:49.872841 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.872657 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-oauth-serving-cert\") pod \"771b1e14-9301-49a6-a09a-ce19ff1871b9\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " Jul 14 11:41:49.872841 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.872694 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr4bj\" (UniqueName: \"kubernetes.io/projected/771b1e14-9301-49a6-a09a-ce19ff1871b9-kube-api-access-wr4bj\") pod \"771b1e14-9301-49a6-a09a-ce19ff1871b9\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " Jul 14 11:41:49.872841 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.872714 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-service-ca\") pod \"771b1e14-9301-49a6-a09a-ce19ff1871b9\" (UID: \"771b1e14-9301-49a6-a09a-ce19ff1871b9\") " Jul 14 11:41:49.873032 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.872983 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-config" (OuterVolumeSpecName: "console-config") pod "771b1e14-9301-49a6-a09a-ce19ff1871b9" (UID: "771b1e14-9301-49a6-a09a-ce19ff1871b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:41:49.873109 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.873037 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "771b1e14-9301-49a6-a09a-ce19ff1871b9" (UID: "771b1e14-9301-49a6-a09a-ce19ff1871b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:41:49.873170 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.873114 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "771b1e14-9301-49a6-a09a-ce19ff1871b9" (UID: "771b1e14-9301-49a6-a09a-ce19ff1871b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:41:49.873227 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.873188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-service-ca" (OuterVolumeSpecName: "service-ca") pod "771b1e14-9301-49a6-a09a-ce19ff1871b9" (UID: "771b1e14-9301-49a6-a09a-ce19ff1871b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:41:49.874750 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.874717 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "771b1e14-9301-49a6-a09a-ce19ff1871b9" (UID: "771b1e14-9301-49a6-a09a-ce19ff1871b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:41:49.874848 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.874817 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771b1e14-9301-49a6-a09a-ce19ff1871b9-kube-api-access-wr4bj" (OuterVolumeSpecName: "kube-api-access-wr4bj") pod "771b1e14-9301-49a6-a09a-ce19ff1871b9" (UID: "771b1e14-9301-49a6-a09a-ce19ff1871b9"). InnerVolumeSpecName "kube-api-access-wr4bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:41:49.874888 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.874864 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "771b1e14-9301-49a6-a09a-ce19ff1871b9" (UID: "771b1e14-9301-49a6-a09a-ce19ff1871b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:41:49.973372 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.973340 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-config\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:49.973372 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.973370 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-serving-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:49.973372 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.973381 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/771b1e14-9301-49a6-a09a-ce19ff1871b9-console-oauth-config\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:49.973525 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.973390 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-trusted-ca-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:49.973525 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.973399 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-oauth-serving-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:49.973525 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.973407 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr4bj\" (UniqueName: \"kubernetes.io/projected/771b1e14-9301-49a6-a09a-ce19ff1871b9-kube-api-access-wr4bj\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:49.973525 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:49.973416 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/771b1e14-9301-49a6-a09a-ce19ff1871b9-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:41:50.540840 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:50.540810 2562 generic.go:356] "Generic (PLEG): container finished" podID="771b1e14-9301-49a6-a09a-ce19ff1871b9" containerID="a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87" exitCode=0 Jul 14 11:41:50.541247 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:50.540854 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdf958b99-82p5g" Jul 14 11:41:50.541247 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:50.540876 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdf958b99-82p5g" event={"ID":"771b1e14-9301-49a6-a09a-ce19ff1871b9","Type":"ContainerDied","Data":"a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87"} Jul 14 11:41:50.541247 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:50.540912 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdf958b99-82p5g" event={"ID":"771b1e14-9301-49a6-a09a-ce19ff1871b9","Type":"ContainerDied","Data":"ebc24ce7db01970d6704d316b06a95638c1c31edf326e0d785b18b6c2505f6d5"} Jul 14 11:41:50.541247 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:50.540934 2562 scope.go:117] "RemoveContainer" containerID="a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87" Jul 14 11:41:50.549907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:50.549888 2562 scope.go:117] "RemoveContainer" containerID="a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87" Jul 14 11:41:50.550220 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:41:50.550200 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87\": container with ID starting with a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87 not found: ID does not exist" containerID="a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87" Jul 14 11:41:50.550284 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:50.550229 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87"} err="failed to get container status \"a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87\": rpc error: code = NotFound desc = could not find container \"a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87\": container with ID starting with a5007b80c8baa852438e2a95b63306d5908dfde54f7fc419afe8a5827136ca87 not found: ID does not exist" Jul 14 11:41:50.570788 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:50.570767 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fdf958b99-82p5g"] Jul 14 11:41:50.580198 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:50.580177 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fdf958b99-82p5g"] Jul 14 11:41:51.479326 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:51.479293 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="771b1e14-9301-49a6-a09a-ce19ff1871b9" path="/var/lib/kubelet/pods/771b1e14-9301-49a6-a09a-ce19ff1871b9/volumes" Jul 14 11:41:55.212253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.212216 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr"] Jul 14 11:41:55.212710 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.212616 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771b1e14-9301-49a6-a09a-ce19ff1871b9" containerName="console" Jul 14 11:41:55.212710 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.212635 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="771b1e14-9301-49a6-a09a-ce19ff1871b9" containerName="console" Jul 14 11:41:55.212819 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.212721 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="771b1e14-9301-49a6-a09a-ce19ff1871b9" containerName="console" Jul 14 11:41:55.217406 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.217383 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.220588 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.220567 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 11:41:55.220707 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.220570 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 11:41:55.220707 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.220616 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-x6gsg\"" Jul 14 11:41:55.226466 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.226437 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr"] Jul 14 11:41:55.307174 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.307151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.307279 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.307224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcb64\" (UniqueName: \"kubernetes.io/projected/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-kube-api-access-zcb64\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.307333 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.307280 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.408567 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.408533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.408567 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.408572 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.408729 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.408703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcb64\" (UniqueName: \"kubernetes.io/projected/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-kube-api-access-zcb64\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.408879 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.408862 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.408948 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.408924 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.418976 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.418950 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcb64\" (UniqueName: \"kubernetes.io/projected/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-kube-api-access-zcb64\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.526551 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.526529 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:41:55.652499 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:55.652478 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr"] Jul 14 11:41:55.655117 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:41:55.655073 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2628bbf8_6638_46f4_a63e_9cf539b6d9e0.slice/crio-1411a96c627518c9e2ed06777b44e71d7886e97f5c0852edb8ba7242df29235f WatchSource:0}: Error finding container 1411a96c627518c9e2ed06777b44e71d7886e97f5c0852edb8ba7242df29235f: Status 404 returned error can't find the container with id 1411a96c627518c9e2ed06777b44e71d7886e97f5c0852edb8ba7242df29235f Jul 14 11:41:56.564827 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:41:56.564788 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" event={"ID":"2628bbf8-6638-46f4-a63e-9cf539b6d9e0","Type":"ContainerStarted","Data":"1411a96c627518c9e2ed06777b44e71d7886e97f5c0852edb8ba7242df29235f"} Jul 14 11:42:03.591710 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:03.591681 2562 generic.go:356] "Generic (PLEG): container finished" podID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerID="956e543bb2f1cd791e7224ee9d1c9ea777a889c8246bdf72e91b1681f7eefc0d" exitCode=0 Jul 14 11:42:03.592012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:03.591718 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" event={"ID":"2628bbf8-6638-46f4-a63e-9cf539b6d9e0","Type":"ContainerDied","Data":"956e543bb2f1cd791e7224ee9d1c9ea777a889c8246bdf72e91b1681f7eefc0d"} Jul 14 11:42:05.602316 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:05.602279 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" event={"ID":"2628bbf8-6638-46f4-a63e-9cf539b6d9e0","Type":"ContainerStarted","Data":"eedf42709c8cba7db700ceffb2edd643e27ec4533ba1479dcb34a30ca5e5e0e7"} Jul 14 11:42:06.608738 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:06.608704 2562 generic.go:356] "Generic (PLEG): container finished" podID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerID="eedf42709c8cba7db700ceffb2edd643e27ec4533ba1479dcb34a30ca5e5e0e7" exitCode=0 Jul 14 11:42:06.609137 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:06.608790 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" event={"ID":"2628bbf8-6638-46f4-a63e-9cf539b6d9e0","Type":"ContainerDied","Data":"eedf42709c8cba7db700ceffb2edd643e27ec4533ba1479dcb34a30ca5e5e0e7"} Jul 14 11:42:14.644110 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:14.644056 2562 generic.go:356] "Generic (PLEG): container finished" podID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerID="e7b8dd8eccde273f721dbf324b4b711808153f42537baff831a3899db077e34e" exitCode=0 Jul 14 11:42:14.644541 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:14.644114 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" event={"ID":"2628bbf8-6638-46f4-a63e-9cf539b6d9e0","Type":"ContainerDied","Data":"e7b8dd8eccde273f721dbf324b4b711808153f42537baff831a3899db077e34e"} Jul 14 11:42:15.764983 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.764962 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:42:15.864151 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.864121 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-util\") pod \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " Jul 14 11:42:15.864282 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.864259 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-bundle\") pod \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " Jul 14 11:42:15.864370 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.864357 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcb64\" (UniqueName: \"kubernetes.io/projected/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-kube-api-access-zcb64\") pod \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\" (UID: \"2628bbf8-6638-46f4-a63e-9cf539b6d9e0\") " Jul 14 11:42:15.864946 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.864926 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-bundle" (OuterVolumeSpecName: "bundle") pod "2628bbf8-6638-46f4-a63e-9cf539b6d9e0" (UID: "2628bbf8-6638-46f4-a63e-9cf539b6d9e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:42:15.866356 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.866332 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-kube-api-access-zcb64" (OuterVolumeSpecName: "kube-api-access-zcb64") pod "2628bbf8-6638-46f4-a63e-9cf539b6d9e0" (UID: "2628bbf8-6638-46f4-a63e-9cf539b6d9e0"). InnerVolumeSpecName "kube-api-access-zcb64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:42:15.867863 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.867836 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-util" (OuterVolumeSpecName: "util") pod "2628bbf8-6638-46f4-a63e-9cf539b6d9e0" (UID: "2628bbf8-6638-46f4-a63e-9cf539b6d9e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:42:15.965841 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.965781 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:42:15.965841 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.965811 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcb64\" (UniqueName: \"kubernetes.io/projected/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-kube-api-access-zcb64\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:42:15.965841 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:15.965822 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2628bbf8-6638-46f4-a63e-9cf539b6d9e0-util\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:42:16.653584 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:16.653557 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" Jul 14 11:42:16.653584 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:16.653572 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540splnr" event={"ID":"2628bbf8-6638-46f4-a63e-9cf539b6d9e0","Type":"ContainerDied","Data":"1411a96c627518c9e2ed06777b44e71d7886e97f5c0852edb8ba7242df29235f"} Jul 14 11:42:16.653766 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:16.653609 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1411a96c627518c9e2ed06777b44e71d7886e97f5c0852edb8ba7242df29235f" Jul 14 11:42:23.730466 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.730438 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd"] Jul 14 11:42:23.730906 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.730689 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerName="util" Jul 14 11:42:23.730906 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.730699 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerName="util" Jul 14 11:42:23.730906 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.730711 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerName="pull" Jul 14 11:42:23.730906 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.730716 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerName="pull" Jul 14 11:42:23.730906 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.730721 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerName="extract" Jul 14 11:42:23.730906 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.730726 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerName="extract" Jul 14 11:42:23.730906 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.730766 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2628bbf8-6638-46f4-a63e-9cf539b6d9e0" containerName="extract" Jul 14 11:42:23.734247 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.734230 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" Jul 14 11:42:23.737145 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.737121 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pdx66\"" Jul 14 11:42:23.737259 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.737164 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 11:42:23.737259 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.737199 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 11:42:23.749224 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.749202 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd"] Jul 14 11:42:23.821955 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.821927 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5649f671-dbcd-4f94-a7fd-3c89cf25b995-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4nsd\" (UID: \"5649f671-dbcd-4f94-a7fd-3c89cf25b995\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" Jul 14 11:42:23.822101 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.821988 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vpv\" (UniqueName: \"kubernetes.io/projected/5649f671-dbcd-4f94-a7fd-3c89cf25b995-kube-api-access-49vpv\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4nsd\" (UID: \"5649f671-dbcd-4f94-a7fd-3c89cf25b995\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" Jul 14 11:42:23.923155 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.923119 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vpv\" (UniqueName: \"kubernetes.io/projected/5649f671-dbcd-4f94-a7fd-3c89cf25b995-kube-api-access-49vpv\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4nsd\" (UID: \"5649f671-dbcd-4f94-a7fd-3c89cf25b995\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" Jul 14 11:42:23.923271 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.923183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5649f671-dbcd-4f94-a7fd-3c89cf25b995-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4nsd\" (UID: \"5649f671-dbcd-4f94-a7fd-3c89cf25b995\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" Jul 14 11:42:23.923506 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.923486 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5649f671-dbcd-4f94-a7fd-3c89cf25b995-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4nsd\" (UID: \"5649f671-dbcd-4f94-a7fd-3c89cf25b995\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" Jul 14 11:42:23.934141 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:23.934122 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vpv\" (UniqueName: \"kubernetes.io/projected/5649f671-dbcd-4f94-a7fd-3c89cf25b995-kube-api-access-49vpv\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4nsd\" (UID: \"5649f671-dbcd-4f94-a7fd-3c89cf25b995\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" Jul 14 11:42:24.042934 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:24.042882 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" Jul 14 11:42:24.167345 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:24.167314 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd"] Jul 14 11:42:24.169913 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:42:24.169890 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5649f671_dbcd_4f94_a7fd_3c89cf25b995.slice/crio-ef063664a95f264631ee7a9e5136931ec0a8f9a2e1056007074da5926659b591 WatchSource:0}: Error finding container ef063664a95f264631ee7a9e5136931ec0a8f9a2e1056007074da5926659b591: Status 404 returned error can't find the container with id ef063664a95f264631ee7a9e5136931ec0a8f9a2e1056007074da5926659b591 Jul 14 11:42:24.683752 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:24.683710 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" event={"ID":"5649f671-dbcd-4f94-a7fd-3c89cf25b995","Type":"ContainerStarted","Data":"ef063664a95f264631ee7a9e5136931ec0a8f9a2e1056007074da5926659b591"} Jul 14 11:42:26.699517 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:26.699477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" event={"ID":"5649f671-dbcd-4f94-a7fd-3c89cf25b995","Type":"ContainerStarted","Data":"ca31dc414ac540aab7b7ad472d88d19156452b7e2a3ace9633e592f08a1eaeb2"} Jul 14 11:42:26.724751 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:26.724709 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4nsd" podStartSLOduration=1.780986683 podStartE2EDuration="3.724695158s" totalImagesPullingTime="1.943708475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:42:26.722420064 +0000 UTC m=+279.798274652" watchObservedRunningTime="2026-07-14 11:42:26.724695158 +0000 UTC m=+279.800549746" Jul 14 11:42:27.639948 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.639912 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds"] Jul 14 11:42:27.642257 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.642232 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.645124 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.645106 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 11:42:27.646397 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.646381 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 11:42:27.646468 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.646410 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-x6gsg\"" Jul 14 11:42:27.653512 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.653486 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds"] Jul 14 11:42:27.755391 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.755362 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.755721 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.755401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.755721 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.755426 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxdwn\" (UniqueName: \"kubernetes.io/projected/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-kube-api-access-bxdwn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.856399 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.856369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.856509 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.856407 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.856509 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.856434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxdwn\" (UniqueName: \"kubernetes.io/projected/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-kube-api-access-bxdwn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.856787 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.856764 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.856845 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.856788 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.866628 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.866601 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxdwn\" (UniqueName: \"kubernetes.io/projected/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-kube-api-access-bxdwn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:27.950979 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:27.950929 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:28.093973 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:28.093942 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds"] Jul 14 11:42:28.097073 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:42:28.097046 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483bb1f9_6b7a_4bf6_965f_ce25e86cdead.slice/crio-8fc1e8aad92f7f37ad6efc0849d8b6507e9dccaf73d8a26b87c9473948a3e7d8 WatchSource:0}: Error finding container 8fc1e8aad92f7f37ad6efc0849d8b6507e9dccaf73d8a26b87c9473948a3e7d8: Status 404 returned error can't find the container with id 8fc1e8aad92f7f37ad6efc0849d8b6507e9dccaf73d8a26b87c9473948a3e7d8 Jul 14 11:42:28.708726 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:28.708690 2562 generic.go:356] "Generic (PLEG): container finished" podID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerID="4ac6fe47db6f91e8a64aad0e05ae12fab89084a9e6e59c0ec459b555058685a8" exitCode=0 Jul 14 11:42:28.708916 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:28.708745 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" event={"ID":"483bb1f9-6b7a-4bf6-965f-ce25e86cdead","Type":"ContainerDied","Data":"4ac6fe47db6f91e8a64aad0e05ae12fab89084a9e6e59c0ec459b555058685a8"} Jul 14 11:42:28.708916 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:28.708779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" event={"ID":"483bb1f9-6b7a-4bf6-965f-ce25e86cdead","Type":"ContainerStarted","Data":"8fc1e8aad92f7f37ad6efc0849d8b6507e9dccaf73d8a26b87c9473948a3e7d8"} Jul 14 11:42:30.272053 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.272021 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq"] Jul 14 11:42:30.274374 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.274353 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" Jul 14 11:42:30.277137 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.277051 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5cprd\"" Jul 14 11:42:30.277137 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.277124 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 11:42:30.278378 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.278363 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 11:42:30.287740 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.287715 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq"] Jul 14 11:42:30.375602 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.375575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5a9db82-0b6c-473e-9d35-d6d9472c63fa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5zhnq\" (UID: \"b5a9db82-0b6c-473e-9d35-d6d9472c63fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" Jul 14 11:42:30.375710 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.375638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4kbd\" (UniqueName: \"kubernetes.io/projected/b5a9db82-0b6c-473e-9d35-d6d9472c63fa-kube-api-access-f4kbd\") pod \"cert-manager-cainjector-559cdd8c87-5zhnq\" (UID: \"b5a9db82-0b6c-473e-9d35-d6d9472c63fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" Jul 14 11:42:30.476122 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.476095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4kbd\" (UniqueName: \"kubernetes.io/projected/b5a9db82-0b6c-473e-9d35-d6d9472c63fa-kube-api-access-f4kbd\") pod \"cert-manager-cainjector-559cdd8c87-5zhnq\" (UID: \"b5a9db82-0b6c-473e-9d35-d6d9472c63fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" Jul 14 11:42:30.476222 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.476129 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5a9db82-0b6c-473e-9d35-d6d9472c63fa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5zhnq\" (UID: \"b5a9db82-0b6c-473e-9d35-d6d9472c63fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" Jul 14 11:42:30.485987 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.485961 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5a9db82-0b6c-473e-9d35-d6d9472c63fa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5zhnq\" (UID: \"b5a9db82-0b6c-473e-9d35-d6d9472c63fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" Jul 14 11:42:30.486125 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.486109 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4kbd\" (UniqueName: \"kubernetes.io/projected/b5a9db82-0b6c-473e-9d35-d6d9472c63fa-kube-api-access-f4kbd\") pod \"cert-manager-cainjector-559cdd8c87-5zhnq\" (UID: \"b5a9db82-0b6c-473e-9d35-d6d9472c63fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" Jul 14 11:42:30.599394 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.599346 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" Jul 14 11:42:30.725483 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:30.725460 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq"] Jul 14 11:42:30.727573 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:42:30.727547 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5a9db82_0b6c_473e_9d35_d6d9472c63fa.slice/crio-e7c65be755e179dcc9a7a4fa186e0a29e21a56e0bf383c8c76ffbe455ff2fe60 WatchSource:0}: Error finding container e7c65be755e179dcc9a7a4fa186e0a29e21a56e0bf383c8c76ffbe455ff2fe60: Status 404 returned error can't find the container with id e7c65be755e179dcc9a7a4fa186e0a29e21a56e0bf383c8c76ffbe455ff2fe60 Jul 14 11:42:30.962072 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:42:30.962033 2562 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: writing blob: storing blob to file \"/var/tmp/container_images_storage2814633128/2\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/8e/8ed306a9eb2cdc7b5db57788b7a70b8d6dae2ecfc602b7662ac4cebb197a1841?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260714%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260714T114230Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=b8afbc3f8398908eae4305b9e9981add2e14c49bfac0cf00cfae950f01f01da7®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=leader-worker-set----lws-operator-bundle&akamai_signature=exp=1784030250~hmac=996d83eeeb9d7405a2ffb40c51548935b13e59d25e44dbb24342a8fe98f63274\": EOF); artifact err: provided artifact is a container image" image="registry.redhat.io/leader-worker-set/lws-operator-bundle@sha256:f641cf3b2d6659e26eb0ae49c67a3416152a2fbc2f26f490889a2bd169b9ea06" Jul 14 11:42:30.962232 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:42:30.962156 2562 kuberuntime_manager.go:1459] "Unhandled Error" err="init container pull start failed in pod 77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds_openshift-marketplace(483bb1f9-6b7a-4bf6-965f-ce25e86cdead): ErrImagePull: unable to pull image or OCI artifact: pull image err: writing blob: storing blob to file \"/var/tmp/container_images_storage2814633128/2\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/8e/8ed306a9eb2cdc7b5db57788b7a70b8d6dae2ecfc602b7662ac4cebb197a1841?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260714%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260714T114230Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=b8afbc3f8398908eae4305b9e9981add2e14c49bfac0cf00cfae950f01f01da7®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=leader-worker-set----lws-operator-bundle&akamai_signature=exp=1784030250~hmac=996d83eeeb9d7405a2ffb40c51548935b13e59d25e44dbb24342a8fe98f63274\": EOF); artifact err: provided artifact is a container image" logger="UnhandledError" Jul 14 11:42:30.962232 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:42:30.962197 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"pull\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: writing blob: storing blob to file \\\"/var/tmp/container_images_storage2814633128/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/8e/8ed306a9eb2cdc7b5db57788b7a70b8d6dae2ecfc602b7662ac4cebb197a1841?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260714%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260714T114230Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=b8afbc3f8398908eae4305b9e9981add2e14c49bfac0cf00cfae950f01f01da7®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=leader-worker-set----lws-operator-bundle&akamai_signature=exp=1784030250~hmac=996d83eeeb9d7405a2ffb40c51548935b13e59d25e44dbb24342a8fe98f63274\\\": EOF); artifact err: provided artifact is a container image\"" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" podUID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" Jul 14 11:42:31.724459 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:31.724424 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" event={"ID":"b5a9db82-0b6c-473e-9d35-d6d9472c63fa","Type":"ContainerStarted","Data":"e7c65be755e179dcc9a7a4fa186e0a29e21a56e0bf383c8c76ffbe455ff2fe60"} Jul 14 11:42:31.725492 ip-10-0-132-116 kubenswrapper[2562]: E0714 11:42:31.725464 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"pull\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/leader-worker-set/lws-operator-bundle@sha256:f641cf3b2d6659e26eb0ae49c67a3416152a2fbc2f26f490889a2bd169b9ea06\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: writing blob: storing blob to file \\\"/var/tmp/container_images_storage2814633128/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/8e/8ed306a9eb2cdc7b5db57788b7a70b8d6dae2ecfc602b7662ac4cebb197a1841?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260714%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260714T114230Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=b8afbc3f8398908eae4305b9e9981add2e14c49bfac0cf00cfae950f01f01da7®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=leader-worker-set----lws-operator-bundle&akamai_signature=exp=1784030250~hmac=996d83eeeb9d7405a2ffb40c51548935b13e59d25e44dbb24342a8fe98f63274\\\": EOF); artifact err: provided artifact is a container image\"" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" podUID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" Jul 14 11:42:35.743398 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:35.743365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" event={"ID":"b5a9db82-0b6c-473e-9d35-d6d9472c63fa","Type":"ContainerStarted","Data":"1c06e772a925ef1b304316a8e2db7c533e6a4f5aea5db09259156ea866a4fd3b"} Jul 14 11:42:35.763397 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:35.763353 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zhnq" podStartSLOduration=1.404355677 podStartE2EDuration="5.763338592s" totalImagesPullingTime="4.358982915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:42:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:42:35.762607703 +0000 UTC m=+288.838462291" watchObservedRunningTime="2026-07-14 11:42:35.763338592 +0000 UTC m=+288.839193179" Jul 14 11:42:47.288678 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:47.288602 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:42:47.290587 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:47.290557 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:42:47.293149 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:47.293123 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:42:47.294804 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:47.294779 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:42:47.299935 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:47.299913 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 11:42:49.806909 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:49.806872 2562 generic.go:356] "Generic (PLEG): container finished" podID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerID="1fc0fff9a95aeac9a7eea754a2eb0c0fbd9ef2dafa93d583041c7e3953b60eab" exitCode=0 Jul 14 11:42:49.807423 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:49.806954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" event={"ID":"483bb1f9-6b7a-4bf6-965f-ce25e86cdead","Type":"ContainerDied","Data":"1fc0fff9a95aeac9a7eea754a2eb0c0fbd9ef2dafa93d583041c7e3953b60eab"} Jul 14 11:42:50.816887 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:50.816854 2562 generic.go:356] "Generic (PLEG): container finished" podID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerID="fda1594cd6934f9c1ce8d01a0a94ec25e58335505ade44faa5ac586c9afede32" exitCode=0 Jul 14 11:42:50.817300 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:50.816935 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" event={"ID":"483bb1f9-6b7a-4bf6-965f-ce25e86cdead","Type":"ContainerDied","Data":"fda1594cd6934f9c1ce8d01a0a94ec25e58335505ade44faa5ac586c9afede32"} Jul 14 11:42:51.948384 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:51.948365 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:52.030958 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.030923 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxdwn\" (UniqueName: \"kubernetes.io/projected/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-kube-api-access-bxdwn\") pod \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " Jul 14 11:42:52.031121 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.030972 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-util\") pod \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " Jul 14 11:42:52.031121 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.031063 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-bundle\") pod \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\" (UID: \"483bb1f9-6b7a-4bf6-965f-ce25e86cdead\") " Jul 14 11:42:52.031427 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.031402 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-bundle" (OuterVolumeSpecName: "bundle") pod "483bb1f9-6b7a-4bf6-965f-ce25e86cdead" (UID: "483bb1f9-6b7a-4bf6-965f-ce25e86cdead"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:42:52.032985 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.032959 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-kube-api-access-bxdwn" (OuterVolumeSpecName: "kube-api-access-bxdwn") pod "483bb1f9-6b7a-4bf6-965f-ce25e86cdead" (UID: "483bb1f9-6b7a-4bf6-965f-ce25e86cdead"). InnerVolumeSpecName "kube-api-access-bxdwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:42:52.035789 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.035765 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-util" (OuterVolumeSpecName: "util") pod "483bb1f9-6b7a-4bf6-965f-ce25e86cdead" (UID: "483bb1f9-6b7a-4bf6-965f-ce25e86cdead"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:42:52.131944 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.131893 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxdwn\" (UniqueName: \"kubernetes.io/projected/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-kube-api-access-bxdwn\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:42:52.131944 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.131915 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-util\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:42:52.131944 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.131924 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/483bb1f9-6b7a-4bf6-965f-ce25e86cdead-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:42:52.827283 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.827261 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" Jul 14 11:42:52.827419 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.827276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgcdds" event={"ID":"483bb1f9-6b7a-4bf6-965f-ce25e86cdead","Type":"ContainerDied","Data":"8fc1e8aad92f7f37ad6efc0849d8b6507e9dccaf73d8a26b87c9473948a3e7d8"} Jul 14 11:42:52.827419 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:52.827321 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fc1e8aad92f7f37ad6efc0849d8b6507e9dccaf73d8a26b87c9473948a3e7d8" Jul 14 11:42:58.663638 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.663603 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj"] Jul 14 11:42:58.664012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.663876 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerName="pull" Jul 14 11:42:58.664012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.663886 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerName="pull" Jul 14 11:42:58.664012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.663901 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerName="util" Jul 14 11:42:58.664012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.663906 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerName="util" Jul 14 11:42:58.664012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.663917 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerName="extract" Jul 14 11:42:58.664012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.663922 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerName="extract" Jul 14 11:42:58.664012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.663972 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="483bb1f9-6b7a-4bf6-965f-ce25e86cdead" containerName="extract" Jul 14 11:42:58.665821 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.665805 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" Jul 14 11:42:58.668673 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.668651 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 11:42:58.670114 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.670093 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 11:42:58.670187 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.670116 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-f4xf8\"" Jul 14 11:42:58.673275 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.673249 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ffbe918-4d30-449d-b0d7-1b8e63415230-tmp\") pod \"openshift-lws-operator-fd8ccff4c-c8wvj\" (UID: \"7ffbe918-4d30-449d-b0d7-1b8e63415230\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" Jul 14 11:42:58.673376 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.673312 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxwj\" (UniqueName: \"kubernetes.io/projected/7ffbe918-4d30-449d-b0d7-1b8e63415230-kube-api-access-hwxwj\") pod \"openshift-lws-operator-fd8ccff4c-c8wvj\" (UID: \"7ffbe918-4d30-449d-b0d7-1b8e63415230\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" Jul 14 11:42:58.674805 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.674781 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj"] Jul 14 11:42:58.774058 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.774033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxwj\" (UniqueName: \"kubernetes.io/projected/7ffbe918-4d30-449d-b0d7-1b8e63415230-kube-api-access-hwxwj\") pod \"openshift-lws-operator-fd8ccff4c-c8wvj\" (UID: \"7ffbe918-4d30-449d-b0d7-1b8e63415230\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" Jul 14 11:42:58.774161 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.774110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ffbe918-4d30-449d-b0d7-1b8e63415230-tmp\") pod \"openshift-lws-operator-fd8ccff4c-c8wvj\" (UID: \"7ffbe918-4d30-449d-b0d7-1b8e63415230\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" Jul 14 11:42:58.774403 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.774388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ffbe918-4d30-449d-b0d7-1b8e63415230-tmp\") pod \"openshift-lws-operator-fd8ccff4c-c8wvj\" (UID: \"7ffbe918-4d30-449d-b0d7-1b8e63415230\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" Jul 14 11:42:58.784567 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.784544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxwj\" (UniqueName: \"kubernetes.io/projected/7ffbe918-4d30-449d-b0d7-1b8e63415230-kube-api-access-hwxwj\") pod \"openshift-lws-operator-fd8ccff4c-c8wvj\" (UID: \"7ffbe918-4d30-449d-b0d7-1b8e63415230\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" Jul 14 11:42:58.978531 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:58.978470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" Jul 14 11:42:59.102172 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:59.102145 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj"] Jul 14 11:42:59.104648 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:42:59.104613 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ffbe918_4d30_449d_b0d7_1b8e63415230.slice/crio-710ce040aaad903fbeed6c91fa1578130991873a21af4b3099f2a4097fb1ad41 WatchSource:0}: Error finding container 710ce040aaad903fbeed6c91fa1578130991873a21af4b3099f2a4097fb1ad41: Status 404 returned error can't find the container with id 710ce040aaad903fbeed6c91fa1578130991873a21af4b3099f2a4097fb1ad41 Jul 14 11:42:59.106015 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:59.105988 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 11:42:59.858211 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:42:59.858169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" event={"ID":"7ffbe918-4d30-449d-b0d7-1b8e63415230","Type":"ContainerStarted","Data":"710ce040aaad903fbeed6c91fa1578130991873a21af4b3099f2a4097fb1ad41"} Jul 14 11:43:01.867777 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:01.867737 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" event={"ID":"7ffbe918-4d30-449d-b0d7-1b8e63415230","Type":"ContainerStarted","Data":"f8d3a4a6e13ed0aae8571ea6a78524264c73bfa99144fa695868d715116f6d34"} Jul 14 11:43:01.898649 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:01.898598 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c8wvj" podStartSLOduration=2.15646938 podStartE2EDuration="3.898583332s" totalImagesPullingTime="1.742113952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:01.898028461 +0000 UTC m=+314.973883045" watchObservedRunningTime="2026-07-14 11:43:01.898583332 +0000 UTC m=+314.974437919" Jul 14 11:43:04.559393 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.559356 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w"] Jul 14 11:43:04.562098 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.562047 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.565119 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.565096 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 11:43:04.565227 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.565143 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 11:43:04.566527 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.566507 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-x6gsg\"" Jul 14 11:43:04.572527 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.572501 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w"] Jul 14 11:43:04.622248 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.622223 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb682\" (UniqueName: \"kubernetes.io/projected/83ded8de-8d42-46b8-b995-06bb585b6d2d-kube-api-access-lb682\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.622337 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.622253 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.622337 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.622300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.722686 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.722654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.722827 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.722706 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb682\" (UniqueName: \"kubernetes.io/projected/83ded8de-8d42-46b8-b995-06bb585b6d2d-kube-api-access-lb682\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.722827 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.722741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.723063 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.723043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.723168 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.723092 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.735792 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.735764 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb682\" (UniqueName: \"kubernetes.io/projected/83ded8de-8d42-46b8-b995-06bb585b6d2d-kube-api-access-lb682\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:04.872448 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:04.872395 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:05.005140 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:05.005107 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w"] Jul 14 11:43:05.007132 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:43:05.007102 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ded8de_8d42_46b8_b995_06bb585b6d2d.slice/crio-cbeb6b3911c9892a5a666cea94800e745d8885127a28af1e57e779f72eaf57dd WatchSource:0}: Error finding container cbeb6b3911c9892a5a666cea94800e745d8885127a28af1e57e779f72eaf57dd: Status 404 returned error can't find the container with id cbeb6b3911c9892a5a666cea94800e745d8885127a28af1e57e779f72eaf57dd Jul 14 11:43:05.891673 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:05.891642 2562 generic.go:356] "Generic (PLEG): container finished" podID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerID="5869fe930b7cc570f6274e676b36ccdd92d9a094b7b2be180464e8bcd459028a" exitCode=0 Jul 14 11:43:05.892019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:05.891704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" event={"ID":"83ded8de-8d42-46b8-b995-06bb585b6d2d","Type":"ContainerDied","Data":"5869fe930b7cc570f6274e676b36ccdd92d9a094b7b2be180464e8bcd459028a"} Jul 14 11:43:05.892019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:05.891733 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" event={"ID":"83ded8de-8d42-46b8-b995-06bb585b6d2d","Type":"ContainerStarted","Data":"cbeb6b3911c9892a5a666cea94800e745d8885127a28af1e57e779f72eaf57dd"} Jul 14 11:43:06.897869 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:06.897840 2562 generic.go:356] "Generic (PLEG): container finished" podID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerID="8764358c8d17f529c02b4e387eff89804bdc244f8a72a7e58327323762829b26" exitCode=0 Jul 14 11:43:06.898282 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:06.897886 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" event={"ID":"83ded8de-8d42-46b8-b995-06bb585b6d2d","Type":"ContainerDied","Data":"8764358c8d17f529c02b4e387eff89804bdc244f8a72a7e58327323762829b26"} Jul 14 11:43:07.903792 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:07.903760 2562 generic.go:356] "Generic (PLEG): container finished" podID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerID="2026daf027d2cbda62206d1490af873a96f6652fe140c5023227680833c9f78f" exitCode=0 Jul 14 11:43:07.904162 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:07.903841 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" event={"ID":"83ded8de-8d42-46b8-b995-06bb585b6d2d","Type":"ContainerDied","Data":"2026daf027d2cbda62206d1490af873a96f6652fe140c5023227680833c9f78f"} Jul 14 11:43:09.026950 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.026930 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:09.054922 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.054896 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-util\") pod \"83ded8de-8d42-46b8-b995-06bb585b6d2d\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " Jul 14 11:43:09.055036 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.054944 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb682\" (UniqueName: \"kubernetes.io/projected/83ded8de-8d42-46b8-b995-06bb585b6d2d-kube-api-access-lb682\") pod \"83ded8de-8d42-46b8-b995-06bb585b6d2d\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " Jul 14 11:43:09.055036 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.054971 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-bundle\") pod \"83ded8de-8d42-46b8-b995-06bb585b6d2d\" (UID: \"83ded8de-8d42-46b8-b995-06bb585b6d2d\") " Jul 14 11:43:09.055825 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.055798 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-bundle" (OuterVolumeSpecName: "bundle") pod "83ded8de-8d42-46b8-b995-06bb585b6d2d" (UID: "83ded8de-8d42-46b8-b995-06bb585b6d2d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:43:09.056962 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.056941 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ded8de-8d42-46b8-b995-06bb585b6d2d-kube-api-access-lb682" (OuterVolumeSpecName: "kube-api-access-lb682") pod "83ded8de-8d42-46b8-b995-06bb585b6d2d" (UID: "83ded8de-8d42-46b8-b995-06bb585b6d2d"). InnerVolumeSpecName "kube-api-access-lb682". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:43:09.063113 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.063090 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-util" (OuterVolumeSpecName: "util") pod "83ded8de-8d42-46b8-b995-06bb585b6d2d" (UID: "83ded8de-8d42-46b8-b995-06bb585b6d2d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:43:09.160650 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.160594 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-util\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:43:09.160808 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.160792 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb682\" (UniqueName: \"kubernetes.io/projected/83ded8de-8d42-46b8-b995-06bb585b6d2d-kube-api-access-lb682\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:43:09.160944 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.160928 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ded8de-8d42-46b8-b995-06bb585b6d2d-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:43:09.913985 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.913926 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" event={"ID":"83ded8de-8d42-46b8-b995-06bb585b6d2d","Type":"ContainerDied","Data":"cbeb6b3911c9892a5a666cea94800e745d8885127a28af1e57e779f72eaf57dd"} Jul 14 11:43:09.913985 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.913952 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8d5w" Jul 14 11:43:09.913985 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:09.913972 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbeb6b3911c9892a5a666cea94800e745d8885127a28af1e57e779f72eaf57dd" Jul 14 11:43:19.130450 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.130418 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp"] Jul 14 11:43:19.130884 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.130701 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerName="pull" Jul 14 11:43:19.130884 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.130712 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerName="pull" Jul 14 11:43:19.130884 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.130724 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerName="util" Jul 14 11:43:19.130884 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.130729 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerName="util" Jul 14 11:43:19.130884 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.130742 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerName="extract" Jul 14 11:43:19.130884 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.130748 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerName="extract" Jul 14 11:43:19.130884 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.130794 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ded8de-8d42-46b8-b995-06bb585b6d2d" containerName="extract" Jul 14 11:43:19.132721 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.132703 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.150872 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.150852 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 11:43:19.152269 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.152246 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 11:43:19.152443 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.152430 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 11:43:19.157048 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.157033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wscvk\"" Jul 14 11:43:19.228752 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.228724 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qr7j\" (UniqueName: \"kubernetes.io/projected/9711ecf6-5274-45b6-965a-e622fb248319-kube-api-access-9qr7j\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.228840 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.228780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9711ecf6-5274-45b6-965a-e622fb248319-metrics-cert\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.228896 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.228867 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9711ecf6-5274-45b6-965a-e622fb248319-cert\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.228942 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.228904 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9711ecf6-5274-45b6-965a-e622fb248319-manager-config\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.239383 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.239366 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp"] Jul 14 11:43:19.329912 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.329885 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9711ecf6-5274-45b6-965a-e622fb248319-metrics-cert\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.330021 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.329928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9711ecf6-5274-45b6-965a-e622fb248319-cert\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.330021 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.329948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9711ecf6-5274-45b6-965a-e622fb248319-manager-config\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.330021 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.329978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qr7j\" (UniqueName: \"kubernetes.io/projected/9711ecf6-5274-45b6-965a-e622fb248319-kube-api-access-9qr7j\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.330648 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.330627 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9711ecf6-5274-45b6-965a-e622fb248319-manager-config\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.332394 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.332375 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9711ecf6-5274-45b6-965a-e622fb248319-metrics-cert\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.332476 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.332429 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9711ecf6-5274-45b6-965a-e622fb248319-cert\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.345721 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.345699 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qr7j\" (UniqueName: \"kubernetes.io/projected/9711ecf6-5274-45b6-965a-e622fb248319-kube-api-access-9qr7j\") pod \"lws-controller-manager-6ff9db667-nx6gp\" (UID: \"9711ecf6-5274-45b6-965a-e622fb248319\") " pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.441284 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.441227 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:19.571326 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:43:19.571300 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9711ecf6_5274_45b6_965a_e622fb248319.slice/crio-a9c248a4033635f24c3fe9dad8c16b0f8c205602cffe0aa781d438859babab83 WatchSource:0}: Error finding container a9c248a4033635f24c3fe9dad8c16b0f8c205602cffe0aa781d438859babab83: Status 404 returned error can't find the container with id a9c248a4033635f24c3fe9dad8c16b0f8c205602cffe0aa781d438859babab83 Jul 14 11:43:19.571905 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.571865 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp"] Jul 14 11:43:19.961744 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:19.961692 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" event={"ID":"9711ecf6-5274-45b6-965a-e622fb248319","Type":"ContainerStarted","Data":"a9c248a4033635f24c3fe9dad8c16b0f8c205602cffe0aa781d438859babab83"} Jul 14 11:43:20.969123 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:20.969061 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" event={"ID":"9711ecf6-5274-45b6-965a-e622fb248319","Type":"ContainerStarted","Data":"dddec4611e34595ca5a8b0410a03b94ad69407f2f6fea6b73d73def20fd18d6f"} Jul 14 11:43:20.969488 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:20.969145 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:20.992147 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:20.992096 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" podStartSLOduration=0.683170439 podStartE2EDuration="1.992064024s" totalImagesPullingTime="1.308893585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:20.990961844 +0000 UTC m=+334.066816432" watchObservedRunningTime="2026-07-14 11:43:20.992064024 +0000 UTC m=+334.067918611" Jul 14 11:43:21.943858 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:21.943825 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct"] Jul 14 11:43:21.946204 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:21.946186 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:21.949539 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:21.949504 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 11:43:21.949539 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:21.949507 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rfn8t\"" Jul 14 11:43:21.949730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:21.949550 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 11:43:21.949730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:21.949554 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 11:43:21.949854 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:21.949840 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 11:43:21.970014 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:21.969988 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct"] Jul 14 11:43:22.051672 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.051641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4bc000c4-a69a-4777-90f5-36ec02e03702-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.051797 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.051756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4bc000c4-a69a-4777-90f5-36ec02e03702-manifests\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.051843 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.051824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bc000c4-a69a-4777-90f5-36ec02e03702-tmp\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.051894 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.051880 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4bc000c4-a69a-4777-90f5-36ec02e03702-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.051941 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.051916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfc5\" (UniqueName: \"kubernetes.io/projected/4bc000c4-a69a-4777-90f5-36ec02e03702-kube-api-access-rhfc5\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.152696 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.152673 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4bc000c4-a69a-4777-90f5-36ec02e03702-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.152826 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.152704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfc5\" (UniqueName: \"kubernetes.io/projected/4bc000c4-a69a-4777-90f5-36ec02e03702-kube-api-access-rhfc5\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.152826 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.152739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4bc000c4-a69a-4777-90f5-36ec02e03702-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.152826 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.152754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4bc000c4-a69a-4777-90f5-36ec02e03702-manifests\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.152826 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.152788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bc000c4-a69a-4777-90f5-36ec02e03702-tmp\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.153289 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.153260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4bc000c4-a69a-4777-90f5-36ec02e03702-manifests\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.155041 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.155016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bc000c4-a69a-4777-90f5-36ec02e03702-tmp\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.155150 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.155121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4bc000c4-a69a-4777-90f5-36ec02e03702-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.155269 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.155252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4bc000c4-a69a-4777-90f5-36ec02e03702-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.164878 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.164855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfc5\" (UniqueName: \"kubernetes.io/projected/4bc000c4-a69a-4777-90f5-36ec02e03702-kube-api-access-rhfc5\") pod \"opendatahub-operator-controller-manager-6c944d8cbf-f4dct\" (UID: \"4bc000c4-a69a-4777-90f5-36ec02e03702\") " pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.256702 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.256683 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:22.387407 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.387383 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct"] Jul 14 11:43:22.389072 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:43:22.389044 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc000c4_a69a_4777_90f5_36ec02e03702.slice/crio-ed348f2522dbb92e777369d2642155363912e7aca6deabe6cb86d84d923e93b7 WatchSource:0}: Error finding container ed348f2522dbb92e777369d2642155363912e7aca6deabe6cb86d84d923e93b7: Status 404 returned error can't find the container with id ed348f2522dbb92e777369d2642155363912e7aca6deabe6cb86d84d923e93b7 Jul 14 11:43:22.981126 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:22.981062 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" event={"ID":"4bc000c4-a69a-4777-90f5-36ec02e03702","Type":"ContainerStarted","Data":"ed348f2522dbb92e777369d2642155363912e7aca6deabe6cb86d84d923e93b7"} Jul 14 11:43:26.001268 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:26.001238 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" event={"ID":"4bc000c4-a69a-4777-90f5-36ec02e03702","Type":"ContainerStarted","Data":"7d1a9b319cfab96d0fb270f91f63656d10eade48777837ae86cbfa4d82152104"} Jul 14 11:43:27.007763 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:27.007721 2562 generic.go:356] "Generic (PLEG): container finished" podID="4bc000c4-a69a-4777-90f5-36ec02e03702" containerID="7d1a9b319cfab96d0fb270f91f63656d10eade48777837ae86cbfa4d82152104" exitCode=0 Jul 14 11:43:27.008156 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:27.007798 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" event={"ID":"4bc000c4-a69a-4777-90f5-36ec02e03702","Type":"ContainerDied","Data":"7d1a9b319cfab96d0fb270f91f63656d10eade48777837ae86cbfa4d82152104"} Jul 14 11:43:28.014303 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:28.014271 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" event={"ID":"4bc000c4-a69a-4777-90f5-36ec02e03702","Type":"ContainerStarted","Data":"be934516ea9117d1d1373f864ac8524bbbd90a5bff518dfa7043be70e348f936"} Jul 14 11:43:28.014755 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:28.014416 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:28.062831 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:28.062781 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" podStartSLOduration=2.715011015 podStartE2EDuration="7.062766737s" totalImagesPullingTime="3.272417341s" totalInitContainerRuntime="1.075338381s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:28.061645446 +0000 UTC m=+341.137500033" watchObservedRunningTime="2026-07-14 11:43:28.062766737 +0000 UTC m=+341.138621394" Jul 14 11:43:31.975730 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:31.975701 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6ff9db667-nx6gp" Jul 14 11:43:39.020243 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:39.020215 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c944d8cbf-f4dct" Jul 14 11:43:50.098535 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.098506 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv"] Jul 14 11:43:50.101726 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.101706 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.105741 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.105715 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cs7t7\"" Jul 14 11:43:50.105938 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.105917 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 11:43:50.123779 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.123752 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv"] Jul 14 11:43:50.256058 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.256029 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.256277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.256145 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.256277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.256174 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.256277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.256201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.256277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.256220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.256277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.256245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.256277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.256274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.256494 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.256306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hgck\" (UniqueName: \"kubernetes.io/projected/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-kube-api-access-2hgck\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.256494 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.256402 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357161 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357273 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357273 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357273 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357273 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357450 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357282 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357450 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357553 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hgck\" (UniqueName: \"kubernetes.io/projected/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-kube-api-access-2hgck\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357553 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357675 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357654 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357825 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357803 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357896 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357811 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357896 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357868 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.357986 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.357901 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.359638 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.359618 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.359764 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.359747 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.374210 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.374183 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.374318 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.374218 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hgck\" (UniqueName: \"kubernetes.io/projected/0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497-kube-api-access-2hgck\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qkhnv\" (UID: \"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.412234 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.412213 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:50.576395 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:50.576372 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv"] Jul 14 11:43:50.578256 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:43:50.578234 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd2fa0d_6a7a_4c9c_9342_c2ede27a8497.slice/crio-2ae7cd4ea3039f588a1761f32bc33c92b86e0eee0e6937602897082638b6ff1b WatchSource:0}: Error finding container 2ae7cd4ea3039f588a1761f32bc33c92b86e0eee0e6937602897082638b6ff1b: Status 404 returned error can't find the container with id 2ae7cd4ea3039f588a1761f32bc33c92b86e0eee0e6937602897082638b6ff1b Jul 14 11:43:51.113542 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:51.113508 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" event={"ID":"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497","Type":"ContainerStarted","Data":"2ae7cd4ea3039f588a1761f32bc33c92b86e0eee0e6937602897082638b6ff1b"} Jul 14 11:43:52.518978 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.518939 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw"] Jul 14 11:43:52.523769 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.523739 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.527037 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.527013 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 11:43:52.527205 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.527019 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 11:43:52.527205 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.527125 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 11:43:52.527331 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.527225 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jkzwd\"" Jul 14 11:43:52.537059 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.537039 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw"] Jul 14 11:43:52.678152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.678106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/010227ea-94b5-4e23-adce-6efe2c354add-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.678319 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.678196 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/010227ea-94b5-4e23-adce-6efe2c354add-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.678319 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.678224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/010227ea-94b5-4e23-adce-6efe2c354add-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.678319 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.678263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs6lw\" (UniqueName: \"kubernetes.io/projected/010227ea-94b5-4e23-adce-6efe2c354add-kube-api-access-bs6lw\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.678319 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.678308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/010227ea-94b5-4e23-adce-6efe2c354add-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.678502 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.678330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/010227ea-94b5-4e23-adce-6efe2c354add-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.678502 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.678361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010227ea-94b5-4e23-adce-6efe2c354add-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.779131 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.779039 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/010227ea-94b5-4e23-adce-6efe2c354add-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.779277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.779129 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/010227ea-94b5-4e23-adce-6efe2c354add-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.779277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.779153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/010227ea-94b5-4e23-adce-6efe2c354add-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.779277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.779196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs6lw\" (UniqueName: \"kubernetes.io/projected/010227ea-94b5-4e23-adce-6efe2c354add-kube-api-access-bs6lw\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.779277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.779225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/010227ea-94b5-4e23-adce-6efe2c354add-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.779277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.779239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/010227ea-94b5-4e23-adce-6efe2c354add-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.779277 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.779262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010227ea-94b5-4e23-adce-6efe2c354add-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.780255 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.780221 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/010227ea-94b5-4e23-adce-6efe2c354add-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.781907 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.781878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/010227ea-94b5-4e23-adce-6efe2c354add-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.782114 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.782061 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/010227ea-94b5-4e23-adce-6efe2c354add-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.782114 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.782066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/010227ea-94b5-4e23-adce-6efe2c354add-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.782321 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.782299 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/010227ea-94b5-4e23-adce-6efe2c354add-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.793493 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.793463 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs6lw\" (UniqueName: \"kubernetes.io/projected/010227ea-94b5-4e23-adce-6efe2c354add-kube-api-access-bs6lw\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.799199 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.799179 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010227ea-94b5-4e23-adce-6efe2c354add-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2xhjw\" (UID: \"010227ea-94b5-4e23-adce-6efe2c354add\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:52.835047 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:52.835027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:53.391967 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:53.391918 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 11:43:53.392115 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:53.392000 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 11:43:53.392115 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:53.392038 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 11:43:53.500247 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:53.500215 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw"] Jul 14 11:43:53.502280 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:43:53.502258 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010227ea_94b5_4e23_adce_6efe2c354add.slice/crio-c68c97b38252dc2baa22fbd834de4ded76e3aaecec2be1d286711ad4c8d36ac4 WatchSource:0}: Error finding container c68c97b38252dc2baa22fbd834de4ded76e3aaecec2be1d286711ad4c8d36ac4: Status 404 returned error can't find the container with id c68c97b38252dc2baa22fbd834de4ded76e3aaecec2be1d286711ad4c8d36ac4 Jul 14 11:43:54.128828 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:54.128793 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" event={"ID":"0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497","Type":"ContainerStarted","Data":"6f97171e037b4e8c6c9527c4b3f65388cff7b0c5fd491cc928bbfef3dd76db8a"} Jul 14 11:43:54.129983 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:54.129958 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" event={"ID":"010227ea-94b5-4e23-adce-6efe2c354add","Type":"ContainerStarted","Data":"c68c97b38252dc2baa22fbd834de4ded76e3aaecec2be1d286711ad4c8d36ac4"} Jul 14 11:43:54.179158 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:54.179098 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" podStartSLOduration=1.367452285 podStartE2EDuration="4.179062855s" totalImagesPullingTime="2.81161057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:54.176914991 +0000 UTC m=+367.252769580" watchObservedRunningTime="2026-07-14 11:43:54.179062855 +0000 UTC m=+367.254917445" Jul 14 11:43:54.412442 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:54.412365 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:54.418342 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:54.418315 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:55.135574 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:55.135542 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:55.136751 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:55.136726 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qkhnv" Jul 14 11:43:56.340650 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:56.340611 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 11:43:56.340940 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:56.340695 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 11:43:57.146129 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:57.146092 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" event={"ID":"010227ea-94b5-4e23-adce-6efe2c354add","Type":"ContainerStarted","Data":"1f4dbaca08a43fc94ebd26423ad246a0d15f8f94f33ce7b5ec521a973c086b19"} Jul 14 11:43:57.146306 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:57.146157 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:43:57.147894 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:57.147869 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-2xhjw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 11:43:57.148008 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:57.147911 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" podUID="010227ea-94b5-4e23-adce-6efe2c354add" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 11:43:57.177011 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:57.176967 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" podStartSLOduration=2.340764703 podStartE2EDuration="5.176953728s" totalImagesPullingTime="2.836189025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:43:57.176138369 +0000 UTC m=+370.251992959" watchObservedRunningTime="2026-07-14 11:43:57.176953728 +0000 UTC m=+370.252808316" Jul 14 11:43:58.152131 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:43:58.152098 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2xhjw" Jul 14 11:44:57.153759 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.153723 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96"] Jul 14 11:44:57.156310 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.156292 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.159203 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.159178 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dbmqg\"" Jul 14 11:44:57.159342 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.159329 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 11:44:57.160614 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.160598 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 11:44:57.167022 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.166999 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96"] Jul 14 11:44:57.246522 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.246497 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.246632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.246526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.246632 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.246552 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lshp\" (UniqueName: \"kubernetes.io/projected/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-kube-api-access-5lshp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.346881 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.346852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.346992 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.346884 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.346992 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.346906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lshp\" (UniqueName: \"kubernetes.io/projected/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-kube-api-access-5lshp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.347341 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.347319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.347341 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.347333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.357169 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.357140 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lshp\" (UniqueName: \"kubernetes.io/projected/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-kube-api-access-5lshp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.466942 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.466884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:44:57.603595 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.603569 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96"] Jul 14 11:44:57.605365 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:44:57.605341 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf9c174_5b84_43bb_b9f5_4c12d25787cd.slice/crio-8f2bec45cd767fbf746d68c1208ca6be97e606bc56efa66cbd271e6de7c0783f WatchSource:0}: Error finding container 8f2bec45cd767fbf746d68c1208ca6be97e606bc56efa66cbd271e6de7c0783f: Status 404 returned error can't find the container with id 8f2bec45cd767fbf746d68c1208ca6be97e606bc56efa66cbd271e6de7c0783f Jul 14 11:44:57.767234 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.767205 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg"] Jul 14 11:44:57.769559 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.769544 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:57.784335 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.784314 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg"] Jul 14 11:44:57.851222 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.851198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvd5m\" (UniqueName: \"kubernetes.io/projected/79b538fa-1efc-4c29-9ea8-504d4ad0bace-kube-api-access-jvd5m\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:57.851311 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.851243 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:57.851311 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.851272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:57.952012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.951988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvd5m\" (UniqueName: \"kubernetes.io/projected/79b538fa-1efc-4c29-9ea8-504d4ad0bace-kube-api-access-jvd5m\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:57.952147 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.952038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:57.952147 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.952067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:57.952428 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.952407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:57.952484 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.952444 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:57.962814 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:57.962790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvd5m\" (UniqueName: \"kubernetes.io/projected/79b538fa-1efc-4c29-9ea8-504d4ad0bace-kube-api-access-jvd5m\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:58.077888 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.077865 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:44:58.215844 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.215813 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg"] Jul 14 11:44:58.218507 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:44:58.218474 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b538fa_1efc_4c29_9ea8_504d4ad0bace.slice/crio-e9fa5947f2b85e480258b140124f90afc07ccdf45412f2ec9805bdcb2ce5cc68 WatchSource:0}: Error finding container e9fa5947f2b85e480258b140124f90afc07ccdf45412f2ec9805bdcb2ce5cc68: Status 404 returned error can't find the container with id e9fa5947f2b85e480258b140124f90afc07ccdf45412f2ec9805bdcb2ce5cc68 Jul 14 11:44:58.426205 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.426171 2562 generic.go:356] "Generic (PLEG): container finished" podID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerID="c429e675c4099af18e7a06dbb2f98cfea2c389b4b671a71c757c1e91522fafc2" exitCode=0 Jul 14 11:44:58.426341 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.426241 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" event={"ID":"79b538fa-1efc-4c29-9ea8-504d4ad0bace","Type":"ContainerDied","Data":"c429e675c4099af18e7a06dbb2f98cfea2c389b4b671a71c757c1e91522fafc2"} Jul 14 11:44:58.426341 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.426288 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" event={"ID":"79b538fa-1efc-4c29-9ea8-504d4ad0bace","Type":"ContainerStarted","Data":"e9fa5947f2b85e480258b140124f90afc07ccdf45412f2ec9805bdcb2ce5cc68"} Jul 14 11:44:58.428142 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.428118 2562 generic.go:356] "Generic (PLEG): container finished" podID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerID="84089fadad2ea01f0861c48c2a79cfee4224e9b4429ebec15bc9a80242b1fba3" exitCode=0 Jul 14 11:44:58.428244 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.428188 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" event={"ID":"5bf9c174-5b84-43bb-b9f5-4c12d25787cd","Type":"ContainerDied","Data":"84089fadad2ea01f0861c48c2a79cfee4224e9b4429ebec15bc9a80242b1fba3"} Jul 14 11:44:58.428244 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.428225 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" event={"ID":"5bf9c174-5b84-43bb-b9f5-4c12d25787cd","Type":"ContainerStarted","Data":"8f2bec45cd767fbf746d68c1208ca6be97e606bc56efa66cbd271e6de7c0783f"} Jul 14 11:44:58.428927 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.428880 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq"] Jul 14 11:44:58.431259 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.431242 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.469939 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.469917 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq"] Jul 14 11:44:58.557030 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.557005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.557159 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.557045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.557222 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.557183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghdn\" (UniqueName: \"kubernetes.io/projected/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-kube-api-access-mghdn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.657987 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.657937 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghdn\" (UniqueName: \"kubernetes.io/projected/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-kube-api-access-mghdn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.658100 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.657987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.658100 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.658026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.658423 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.658404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.658465 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.658420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.668728 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.668703 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghdn\" (UniqueName: \"kubernetes.io/projected/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-kube-api-access-mghdn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.739737 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.739719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:44:58.864360 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.864334 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq"] Jul 14 11:44:58.865768 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:44:58.865738 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c575d8e_7ae5_4be4_a1e3_af3a6e77aaa9.slice/crio-3c999a250861bc56bc8e60456a2d6a7de57404a93726be168177e2640dbf7dc2 WatchSource:0}: Error finding container 3c999a250861bc56bc8e60456a2d6a7de57404a93726be168177e2640dbf7dc2: Status 404 returned error can't find the container with id 3c999a250861bc56bc8e60456a2d6a7de57404a93726be168177e2640dbf7dc2 Jul 14 11:44:58.963350 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.963324 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9"] Jul 14 11:44:58.965493 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.965473 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:58.976514 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:58.976491 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9"] Jul 14 11:44:59.062296 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.062267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.062419 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.062375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.062419 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.062402 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-485qv\" (UniqueName: \"kubernetes.io/projected/21f46e89-dcda-4bd1-a630-001d517a07a3-kube-api-access-485qv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.162829 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.162811 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.162918 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.162878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.162918 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.162897 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-485qv\" (UniqueName: \"kubernetes.io/projected/21f46e89-dcda-4bd1-a630-001d517a07a3-kube-api-access-485qv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.163222 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.163200 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.163376 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.163358 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.172725 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.172695 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-485qv\" (UniqueName: \"kubernetes.io/projected/21f46e89-dcda-4bd1-a630-001d517a07a3-kube-api-access-485qv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.331065 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.331037 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:44:59.436287 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.436215 2562 generic.go:356] "Generic (PLEG): container finished" podID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerID="256ab266848c04dfe3a3103e552b30d5983317870059408fb75e7f564ddc4b6f" exitCode=0 Jul 14 11:44:59.436425 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.436294 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" event={"ID":"5bf9c174-5b84-43bb-b9f5-4c12d25787cd","Type":"ContainerDied","Data":"256ab266848c04dfe3a3103e552b30d5983317870059408fb75e7f564ddc4b6f"} Jul 14 11:44:59.437985 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.437959 2562 generic.go:356] "Generic (PLEG): container finished" podID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerID="cf8ce93770b254287af98c8cb9875000706efd8f5323db06d9dd218edb02b70b" exitCode=0 Jul 14 11:44:59.438143 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.438037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" event={"ID":"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9","Type":"ContainerDied","Data":"cf8ce93770b254287af98c8cb9875000706efd8f5323db06d9dd218edb02b70b"} Jul 14 11:44:59.438143 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.438106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" event={"ID":"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9","Type":"ContainerStarted","Data":"3c999a250861bc56bc8e60456a2d6a7de57404a93726be168177e2640dbf7dc2"} Jul 14 11:44:59.440253 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.440203 2562 generic.go:356] "Generic (PLEG): container finished" podID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerID="13deb7e106db2da2f80312152f4ec0249e17ea2dd7ec43b5dc21b2d28ac6e1e1" exitCode=0 Jul 14 11:44:59.440343 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.440254 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" event={"ID":"79b538fa-1efc-4c29-9ea8-504d4ad0bace","Type":"ContainerDied","Data":"13deb7e106db2da2f80312152f4ec0249e17ea2dd7ec43b5dc21b2d28ac6e1e1"} Jul 14 11:44:59.459564 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:44:59.459495 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9"] Jul 14 11:44:59.461265 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:44:59.461241 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f46e89_dcda_4bd1_a630_001d517a07a3.slice/crio-ac5887cd3a031a0ac624c67997b85f2c485b0e0f2f061530cdbbdb2f3e5ed761 WatchSource:0}: Error finding container ac5887cd3a031a0ac624c67997b85f2c485b0e0f2f061530cdbbdb2f3e5ed761: Status 404 returned error can't find the container with id ac5887cd3a031a0ac624c67997b85f2c485b0e0f2f061530cdbbdb2f3e5ed761 Jul 14 11:45:00.447119 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:00.447071 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" event={"ID":"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9","Type":"ContainerStarted","Data":"04688c787dc138e4dba16f5aa6b34c7a43db8ebfa4e4b8c6906010ee2e0974eb"} Jul 14 11:45:00.448776 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:00.448750 2562 generic.go:356] "Generic (PLEG): container finished" podID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerID="cd60ce2cda98d60924280b9935ec37dca9a1ae1b302ede5a375c05418a6f1ebb" exitCode=0 Jul 14 11:45:00.448870 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:00.448828 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" event={"ID":"21f46e89-dcda-4bd1-a630-001d517a07a3","Type":"ContainerDied","Data":"cd60ce2cda98d60924280b9935ec37dca9a1ae1b302ede5a375c05418a6f1ebb"} Jul 14 11:45:00.448938 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:00.448871 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" event={"ID":"21f46e89-dcda-4bd1-a630-001d517a07a3","Type":"ContainerStarted","Data":"ac5887cd3a031a0ac624c67997b85f2c485b0e0f2f061530cdbbdb2f3e5ed761"} Jul 14 11:45:00.450855 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:00.450833 2562 generic.go:356] "Generic (PLEG): container finished" podID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerID="91811e9ce45fb04a9f2c3a37fc7618b1351a2c4b8ab563c89a95bc4c25d18f1f" exitCode=0 Jul 14 11:45:00.450999 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:00.450944 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" event={"ID":"79b538fa-1efc-4c29-9ea8-504d4ad0bace","Type":"ContainerDied","Data":"91811e9ce45fb04a9f2c3a37fc7618b1351a2c4b8ab563c89a95bc4c25d18f1f"} Jul 14 11:45:00.453322 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:00.453239 2562 generic.go:356] "Generic (PLEG): container finished" podID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerID="c4aafa8d5a9e717f7d26428a4017ed18534bf98ddae958981dcbc48bba759df4" exitCode=0 Jul 14 11:45:00.453322 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:00.453283 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" event={"ID":"5bf9c174-5b84-43bb-b9f5-4c12d25787cd","Type":"ContainerDied","Data":"c4aafa8d5a9e717f7d26428a4017ed18534bf98ddae958981dcbc48bba759df4"} Jul 14 11:45:01.459608 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.459582 2562 generic.go:356] "Generic (PLEG): container finished" podID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerID="04688c787dc138e4dba16f5aa6b34c7a43db8ebfa4e4b8c6906010ee2e0974eb" exitCode=0 Jul 14 11:45:01.459938 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.459652 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" event={"ID":"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9","Type":"ContainerDied","Data":"04688c787dc138e4dba16f5aa6b34c7a43db8ebfa4e4b8c6906010ee2e0974eb"} Jul 14 11:45:01.461520 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.461499 2562 generic.go:356] "Generic (PLEG): container finished" podID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerID="66691887deee9539c2931fbaef042b2afad2c516b359db84f7b407afc660c367" exitCode=0 Jul 14 11:45:01.461623 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.461576 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" event={"ID":"21f46e89-dcda-4bd1-a630-001d517a07a3","Type":"ContainerDied","Data":"66691887deee9539c2931fbaef042b2afad2c516b359db84f7b407afc660c367"} Jul 14 11:45:01.613877 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.613839 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:45:01.684450 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.684425 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-bundle\") pod \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " Jul 14 11:45:01.684557 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.684476 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvd5m\" (UniqueName: \"kubernetes.io/projected/79b538fa-1efc-4c29-9ea8-504d4ad0bace-kube-api-access-jvd5m\") pod \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " Jul 14 11:45:01.684557 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.684517 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-util\") pod \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\" (UID: \"79b538fa-1efc-4c29-9ea8-504d4ad0bace\") " Jul 14 11:45:01.685002 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.684978 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-bundle" (OuterVolumeSpecName: "bundle") pod "79b538fa-1efc-4c29-9ea8-504d4ad0bace" (UID: "79b538fa-1efc-4c29-9ea8-504d4ad0bace"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:45:01.686509 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.686488 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b538fa-1efc-4c29-9ea8-504d4ad0bace-kube-api-access-jvd5m" (OuterVolumeSpecName: "kube-api-access-jvd5m") pod "79b538fa-1efc-4c29-9ea8-504d4ad0bace" (UID: "79b538fa-1efc-4c29-9ea8-504d4ad0bace"). InnerVolumeSpecName "kube-api-access-jvd5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:45:01.689479 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.689448 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-util" (OuterVolumeSpecName: "util") pod "79b538fa-1efc-4c29-9ea8-504d4ad0bace" (UID: "79b538fa-1efc-4c29-9ea8-504d4ad0bace"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:45:01.719459 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.719438 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:45:01.785299 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.785254 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lshp\" (UniqueName: \"kubernetes.io/projected/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-kube-api-access-5lshp\") pod \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " Jul 14 11:45:01.785380 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.785315 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-bundle\") pod \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " Jul 14 11:45:01.785442 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.785422 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-util\") pod \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\" (UID: \"5bf9c174-5b84-43bb-b9f5-4c12d25787cd\") " Jul 14 11:45:01.785719 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.785690 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:01.785719 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.785708 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvd5m\" (UniqueName: \"kubernetes.io/projected/79b538fa-1efc-4c29-9ea8-504d4ad0bace-kube-api-access-jvd5m\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:01.785719 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.785717 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79b538fa-1efc-4c29-9ea8-504d4ad0bace-util\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:01.785878 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.785737 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-bundle" (OuterVolumeSpecName: "bundle") pod "5bf9c174-5b84-43bb-b9f5-4c12d25787cd" (UID: "5bf9c174-5b84-43bb-b9f5-4c12d25787cd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:45:01.787152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.787134 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-kube-api-access-5lshp" (OuterVolumeSpecName: "kube-api-access-5lshp") pod "5bf9c174-5b84-43bb-b9f5-4c12d25787cd" (UID: "5bf9c174-5b84-43bb-b9f5-4c12d25787cd"). InnerVolumeSpecName "kube-api-access-5lshp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:45:01.790587 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.790556 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-util" (OuterVolumeSpecName: "util") pod "5bf9c174-5b84-43bb-b9f5-4c12d25787cd" (UID: "5bf9c174-5b84-43bb-b9f5-4c12d25787cd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:45:01.886012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.885991 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-util\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:01.886012 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.886012 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lshp\" (UniqueName: \"kubernetes.io/projected/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-kube-api-access-5lshp\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:01.886165 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:01.886023 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bf9c174-5b84-43bb-b9f5-4c12d25787cd-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:02.468056 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.468035 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" Jul 14 11:45:02.468056 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.468042 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96" event={"ID":"5bf9c174-5b84-43bb-b9f5-4c12d25787cd","Type":"ContainerDied","Data":"8f2bec45cd767fbf746d68c1208ca6be97e606bc56efa66cbd271e6de7c0783f"} Jul 14 11:45:02.468519 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.468106 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f2bec45cd767fbf746d68c1208ca6be97e606bc56efa66cbd271e6de7c0783f" Jul 14 11:45:02.470228 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.470204 2562 generic.go:356] "Generic (PLEG): container finished" podID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerID="c8a15eb488648b7e402aae892228f1067151580db603bf6aee149d8b48db1de7" exitCode=0 Jul 14 11:45:02.470332 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.470273 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" event={"ID":"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9","Type":"ContainerDied","Data":"c8a15eb488648b7e402aae892228f1067151580db603bf6aee149d8b48db1de7"} Jul 14 11:45:02.472396 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.472376 2562 generic.go:356] "Generic (PLEG): container finished" podID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerID="253b4a55e73684b6021adc3fb12f8e023b0bd023de3c9efc50a92da7923ee76c" exitCode=0 Jul 14 11:45:02.472491 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.472454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" event={"ID":"21f46e89-dcda-4bd1-a630-001d517a07a3","Type":"ContainerDied","Data":"253b4a55e73684b6021adc3fb12f8e023b0bd023de3c9efc50a92da7923ee76c"} Jul 14 11:45:02.474332 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.474311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" event={"ID":"79b538fa-1efc-4c29-9ea8-504d4ad0bace","Type":"ContainerDied","Data":"e9fa5947f2b85e480258b140124f90afc07ccdf45412f2ec9805bdcb2ce5cc68"} Jul 14 11:45:02.474332 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.474327 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg" Jul 14 11:45:02.474495 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:02.474342 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9fa5947f2b85e480258b140124f90afc07ccdf45412f2ec9805bdcb2ce5cc68" Jul 14 11:45:03.605677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.605654 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:45:03.635908 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.635882 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:45:03.699925 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.699901 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-bundle\") pod \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " Jul 14 11:45:03.700045 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.699928 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-util\") pod \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " Jul 14 11:45:03.700045 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.699964 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-util\") pod \"21f46e89-dcda-4bd1-a630-001d517a07a3\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " Jul 14 11:45:03.700045 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.699986 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mghdn\" (UniqueName: \"kubernetes.io/projected/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-kube-api-access-mghdn\") pod \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\" (UID: \"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9\") " Jul 14 11:45:03.700045 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.700009 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-485qv\" (UniqueName: \"kubernetes.io/projected/21f46e89-dcda-4bd1-a630-001d517a07a3-kube-api-access-485qv\") pod \"21f46e89-dcda-4bd1-a630-001d517a07a3\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " Jul 14 11:45:03.700045 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.700042 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-bundle\") pod \"21f46e89-dcda-4bd1-a630-001d517a07a3\" (UID: \"21f46e89-dcda-4bd1-a630-001d517a07a3\") " Jul 14 11:45:03.700569 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.700543 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-bundle" (OuterVolumeSpecName: "bundle") pod "9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" (UID: "9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:45:03.700637 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.700607 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-bundle" (OuterVolumeSpecName: "bundle") pod "21f46e89-dcda-4bd1-a630-001d517a07a3" (UID: "21f46e89-dcda-4bd1-a630-001d517a07a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:45:03.702466 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.702434 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f46e89-dcda-4bd1-a630-001d517a07a3-kube-api-access-485qv" (OuterVolumeSpecName: "kube-api-access-485qv") pod "21f46e89-dcda-4bd1-a630-001d517a07a3" (UID: "21f46e89-dcda-4bd1-a630-001d517a07a3"). InnerVolumeSpecName "kube-api-access-485qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:45:03.702576 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.702499 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-kube-api-access-mghdn" (OuterVolumeSpecName: "kube-api-access-mghdn") pod "9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" (UID: "9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9"). InnerVolumeSpecName "kube-api-access-mghdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:45:03.706265 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.706233 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-util" (OuterVolumeSpecName: "util") pod "21f46e89-dcda-4bd1-a630-001d517a07a3" (UID: "21f46e89-dcda-4bd1-a630-001d517a07a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:45:03.706443 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.706423 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-util" (OuterVolumeSpecName: "util") pod "9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" (UID: "9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 11:45:03.801354 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.801302 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-util\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:03.801354 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.801324 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghdn\" (UniqueName: \"kubernetes.io/projected/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-kube-api-access-mghdn\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:03.801354 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.801336 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-485qv\" (UniqueName: \"kubernetes.io/projected/21f46e89-dcda-4bd1-a630-001d517a07a3-kube-api-access-485qv\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:03.801354 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.801345 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21f46e89-dcda-4bd1-a630-001d517a07a3-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:03.801354 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.801353 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:03.801547 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:03.801362 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9-util\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:04.486690 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:04.486669 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" Jul 14 11:45:04.486832 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:04.486662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq" event={"ID":"9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9","Type":"ContainerDied","Data":"3c999a250861bc56bc8e60456a2d6a7de57404a93726be168177e2640dbf7dc2"} Jul 14 11:45:04.486832 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:04.486813 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c999a250861bc56bc8e60456a2d6a7de57404a93726be168177e2640dbf7dc2" Jul 14 11:45:04.488762 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:04.488743 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" Jul 14 11:45:04.488855 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:04.488738 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9" event={"ID":"21f46e89-dcda-4bd1-a630-001d517a07a3","Type":"ContainerDied","Data":"ac5887cd3a031a0ac624c67997b85f2c485b0e0f2f061530cdbbdb2f3e5ed761"} Jul 14 11:45:04.488896 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:04.488858 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac5887cd3a031a0ac624c67997b85f2c485b0e0f2f061530cdbbdb2f3e5ed761" Jul 14 11:45:19.479894 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:19.479813 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b767f6bc-6ngmm"] Jul 14 11:45:44.499540 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.499477 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84b767f6bc-6ngmm" podUID="96c1232b-bdcd-4a4b-949b-7b22205f648f" containerName="console" containerID="cri-o://2534a288cf494934cce7c5cc89fc31e5c6d1db2b74f112ebb5caf66850faf400" gracePeriod=15 Jul 14 11:45:44.684571 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.684380 2562 generic.go:356] "Generic (PLEG): container finished" podID="96c1232b-bdcd-4a4b-949b-7b22205f648f" containerID="2534a288cf494934cce7c5cc89fc31e5c6d1db2b74f112ebb5caf66850faf400" exitCode=0 Jul 14 11:45:44.684571 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.684446 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b767f6bc-6ngmm" event={"ID":"96c1232b-bdcd-4a4b-949b-7b22205f648f","Type":"ContainerDied","Data":"2534a288cf494934cce7c5cc89fc31e5c6d1db2b74f112ebb5caf66850faf400"} Jul 14 11:45:44.740884 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.740862 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:45:44.814026 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.813961 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtqsr\" (UniqueName: \"kubernetes.io/projected/96c1232b-bdcd-4a4b-949b-7b22205f648f-kube-api-access-gtqsr\") pod \"96c1232b-bdcd-4a4b-949b-7b22205f648f\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " Jul 14 11:45:44.814026 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.813997 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-service-ca\") pod \"96c1232b-bdcd-4a4b-949b-7b22205f648f\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " Jul 14 11:45:44.814026 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.814021 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-trusted-ca-bundle\") pod \"96c1232b-bdcd-4a4b-949b-7b22205f648f\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " Jul 14 11:45:44.814336 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.814064 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-serving-cert\") pod \"96c1232b-bdcd-4a4b-949b-7b22205f648f\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " Jul 14 11:45:44.814336 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.814121 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-config\") pod \"96c1232b-bdcd-4a4b-949b-7b22205f648f\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " Jul 14 11:45:44.814336 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.814147 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-oauth-config\") pod \"96c1232b-bdcd-4a4b-949b-7b22205f648f\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " Jul 14 11:45:44.814336 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.814194 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-oauth-serving-cert\") pod \"96c1232b-bdcd-4a4b-949b-7b22205f648f\" (UID: \"96c1232b-bdcd-4a4b-949b-7b22205f648f\") " Jul 14 11:45:44.814583 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.814471 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-service-ca" (OuterVolumeSpecName: "service-ca") pod "96c1232b-bdcd-4a4b-949b-7b22205f648f" (UID: "96c1232b-bdcd-4a4b-949b-7b22205f648f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:45:44.814583 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.814506 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-config" (OuterVolumeSpecName: "console-config") pod "96c1232b-bdcd-4a4b-949b-7b22205f648f" (UID: "96c1232b-bdcd-4a4b-949b-7b22205f648f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:45:44.814583 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.814517 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "96c1232b-bdcd-4a4b-949b-7b22205f648f" (UID: "96c1232b-bdcd-4a4b-949b-7b22205f648f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:45:44.814784 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.814763 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "96c1232b-bdcd-4a4b-949b-7b22205f648f" (UID: "96c1232b-bdcd-4a4b-949b-7b22205f648f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 11:45:44.816247 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.816223 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "96c1232b-bdcd-4a4b-949b-7b22205f648f" (UID: "96c1232b-bdcd-4a4b-949b-7b22205f648f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:45:44.816247 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.816236 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "96c1232b-bdcd-4a4b-949b-7b22205f648f" (UID: "96c1232b-bdcd-4a4b-949b-7b22205f648f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:45:44.816371 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.816267 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c1232b-bdcd-4a4b-949b-7b22205f648f-kube-api-access-gtqsr" (OuterVolumeSpecName: "kube-api-access-gtqsr") pod "96c1232b-bdcd-4a4b-949b-7b22205f648f" (UID: "96c1232b-bdcd-4a4b-949b-7b22205f648f"). InnerVolumeSpecName "kube-api-access-gtqsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:45:44.914792 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.914762 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-oauth-serving-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:44.914792 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.914788 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtqsr\" (UniqueName: \"kubernetes.io/projected/96c1232b-bdcd-4a4b-949b-7b22205f648f-kube-api-access-gtqsr\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:44.914934 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.914802 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:44.914934 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.914814 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-trusted-ca-bundle\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:44.914934 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.914825 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-serving-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:44.914934 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.914839 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-config\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:44.914934 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:44.914851 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96c1232b-bdcd-4a4b-949b-7b22205f648f-console-oauth-config\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:45:45.690720 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:45.690694 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b767f6bc-6ngmm" Jul 14 11:45:45.691176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:45.690717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b767f6bc-6ngmm" event={"ID":"96c1232b-bdcd-4a4b-949b-7b22205f648f","Type":"ContainerDied","Data":"7b3caca037e02d59909278702f52becafca2c333e6c8b9e49b7c8d8abd7b4f4d"} Jul 14 11:45:45.691176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:45.690778 2562 scope.go:117] "RemoveContainer" containerID="2534a288cf494934cce7c5cc89fc31e5c6d1db2b74f112ebb5caf66850faf400" Jul 14 11:45:45.719626 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:45.719599 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b767f6bc-6ngmm"] Jul 14 11:45:45.723272 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:45.723250 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84b767f6bc-6ngmm"] Jul 14 11:45:47.478058 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:45:47.478028 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96c1232b-bdcd-4a4b-949b-7b22205f648f" path="/var/lib/kubelet/pods/96c1232b-bdcd-4a4b-949b-7b22205f648f/volumes" Jul 14 11:46:58.942658 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.942627 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c4d6c7474-x6rcn"] Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.942940 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerName="util" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.942952 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerName="util" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.942962 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerName="util" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.942968 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerName="util" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.942977 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerName="pull" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.942982 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerName="pull" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.942993 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerName="extract" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.942998 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerName="extract" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943009 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c1232b-bdcd-4a4b-949b-7b22205f648f" containerName="console" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943014 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c1232b-bdcd-4a4b-949b-7b22205f648f" containerName="console" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943019 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerName="util" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943024 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerName="util" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943029 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerName="pull" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943034 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerName="pull" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943040 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerName="extract" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943044 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerName="extract" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943051 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerName="pull" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943057 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerName="pull" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943064 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerName="util" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943068 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerName="util" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943086 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerName="extract" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943092 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerName="extract" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943098 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerName="pull" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943104 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerName="pull" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943110 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerName="extract" Jul 14 11:46:58.943134 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943115 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerName="extract" Jul 14 11:46:58.943977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943160 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf9c174-5b84-43bb-b9f5-4c12d25787cd" containerName="extract" Jul 14 11:46:58.943977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943168 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f46e89-dcda-4bd1-a630-001d517a07a3" containerName="extract" Jul 14 11:46:58.943977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943174 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b538fa-1efc-4c29-9ea8-504d4ad0bace" containerName="extract" Jul 14 11:46:58.943977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943180 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9" containerName="extract" Jul 14 11:46:58.943977 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.943187 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c1232b-bdcd-4a4b-949b-7b22205f648f" containerName="console" Jul 14 11:46:58.945816 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.945801 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:46:58.950590 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.950554 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 11:46:58.950590 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.950586 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 11:46:58.950763 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.950590 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 11:46:58.950763 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.950611 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rnk6k\"" Jul 14 11:46:58.954599 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:58.954580 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c4d6c7474-x6rcn"] Jul 14 11:46:59.048722 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:59.048697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21dbc61d-fbfe-41ae-8232-b9278badad92-tls-cert\") pod \"authorino-c4d6c7474-x6rcn\" (UID: \"21dbc61d-fbfe-41ae-8232-b9278badad92\") " pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:46:59.048830 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:59.048729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlsgr\" (UniqueName: \"kubernetes.io/projected/21dbc61d-fbfe-41ae-8232-b9278badad92-kube-api-access-zlsgr\") pod \"authorino-c4d6c7474-x6rcn\" (UID: \"21dbc61d-fbfe-41ae-8232-b9278badad92\") " pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:46:59.149048 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:59.149027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21dbc61d-fbfe-41ae-8232-b9278badad92-tls-cert\") pod \"authorino-c4d6c7474-x6rcn\" (UID: \"21dbc61d-fbfe-41ae-8232-b9278badad92\") " pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:46:59.149189 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:59.149054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlsgr\" (UniqueName: \"kubernetes.io/projected/21dbc61d-fbfe-41ae-8232-b9278badad92-kube-api-access-zlsgr\") pod \"authorino-c4d6c7474-x6rcn\" (UID: \"21dbc61d-fbfe-41ae-8232-b9278badad92\") " pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:46:59.151363 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:59.151337 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21dbc61d-fbfe-41ae-8232-b9278badad92-tls-cert\") pod \"authorino-c4d6c7474-x6rcn\" (UID: \"21dbc61d-fbfe-41ae-8232-b9278badad92\") " pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:46:59.162746 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:59.162719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlsgr\" (UniqueName: \"kubernetes.io/projected/21dbc61d-fbfe-41ae-8232-b9278badad92-kube-api-access-zlsgr\") pod \"authorino-c4d6c7474-x6rcn\" (UID: \"21dbc61d-fbfe-41ae-8232-b9278badad92\") " pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:46:59.254805 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:59.254779 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:46:59.385323 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:46:59.380704 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c4d6c7474-x6rcn"] Jul 14 11:47:00.062721 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:00.062664 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" event={"ID":"21dbc61d-fbfe-41ae-8232-b9278badad92","Type":"ContainerStarted","Data":"f41837cbe6a6d6739bdc542c0db1d448468a8eae9c781c2e763b079a071aa31e"} Jul 14 11:47:03.085447 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:03.085408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" event={"ID":"21dbc61d-fbfe-41ae-8232-b9278badad92","Type":"ContainerStarted","Data":"9a674f7d3e51385c5c4d8af8d702e41e9f86e1821fe0f6db171187cd1435ded0"} Jul 14 11:47:03.104413 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:03.104360 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" podStartSLOduration=2.315157823 podStartE2EDuration="5.104345395s" totalImagesPullingTime="2.789187572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:47:03.103293027 +0000 UTC m=+556.179147613" watchObservedRunningTime="2026-07-14 11:47:03.104345395 +0000 UTC m=+556.180199983" Jul 14 11:47:42.870028 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:42.869996 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll"] Jul 14 11:47:42.872152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:42.872136 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:47:42.875648 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:42.875623 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-h6nsc\"" Jul 14 11:47:42.875775 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:42.875667 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 14 11:47:42.884644 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:42.884622 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll"] Jul 14 11:47:42.971153 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:42.971115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6tgf\" (UniqueName: \"kubernetes.io/projected/aaf74151-7087-4688-b78f-713f36f0a271-kube-api-access-j6tgf\") pod \"payload-pre-processing-74584bb9f8-tp2ll\" (UID: \"aaf74151-7087-4688-b78f-713f36f0a271\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:47:42.971330 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:42.971237 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf74151-7087-4688-b78f-713f36f0a271-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-tp2ll\" (UID: \"aaf74151-7087-4688-b78f-713f36f0a271\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:47:43.072266 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:43.072232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tgf\" (UniqueName: \"kubernetes.io/projected/aaf74151-7087-4688-b78f-713f36f0a271-kube-api-access-j6tgf\") pod \"payload-pre-processing-74584bb9f8-tp2ll\" (UID: \"aaf74151-7087-4688-b78f-713f36f0a271\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:47:43.072382 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:43.072274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf74151-7087-4688-b78f-713f36f0a271-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-tp2ll\" (UID: \"aaf74151-7087-4688-b78f-713f36f0a271\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:47:43.072965 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:43.072947 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf74151-7087-4688-b78f-713f36f0a271-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-tp2ll\" (UID: \"aaf74151-7087-4688-b78f-713f36f0a271\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:47:43.082841 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:43.082814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6tgf\" (UniqueName: \"kubernetes.io/projected/aaf74151-7087-4688-b78f-713f36f0a271-kube-api-access-j6tgf\") pod \"payload-pre-processing-74584bb9f8-tp2ll\" (UID: \"aaf74151-7087-4688-b78f-713f36f0a271\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:47:43.181932 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:43.181877 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:47:43.310417 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:43.310390 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll"] Jul 14 11:47:43.312063 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:47:43.312029 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf74151_7087_4688_b78f_713f36f0a271.slice/crio-b635f443b568e0f898efd0d1a1c35f2552205cb7df9492d10b47048a18d0aaaa WatchSource:0}: Error finding container b635f443b568e0f898efd0d1a1c35f2552205cb7df9492d10b47048a18d0aaaa: Status 404 returned error can't find the container with id b635f443b568e0f898efd0d1a1c35f2552205cb7df9492d10b47048a18d0aaaa Jul 14 11:47:44.290757 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:44.290724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" event={"ID":"aaf74151-7087-4688-b78f-713f36f0a271","Type":"ContainerStarted","Data":"b635f443b568e0f898efd0d1a1c35f2552205cb7df9492d10b47048a18d0aaaa"} Jul 14 11:47:45.297035 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:45.297003 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" event={"ID":"aaf74151-7087-4688-b78f-713f36f0a271","Type":"ContainerStarted","Data":"d0177bd2a640efdc5636b76fe907da542fe3463440129d99baad90473b83aa8c"} Jul 14 11:47:45.297388 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:45.297137 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:47:45.324474 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:45.324434 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" podStartSLOduration=1.8245863089999999 podStartE2EDuration="3.324421218s" totalImagesPullingTime="1.499834909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:47:45.323952805 +0000 UTC m=+598.399807394" watchObservedRunningTime="2026-07-14 11:47:45.324421218 +0000 UTC m=+598.400275806" Jul 14 11:47:47.323679 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:47.323653 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:47:47.324101 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:47.324024 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:47:47.327027 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:47.327004 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:47:47.327356 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:47.327341 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:47:56.305265 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:47:56.305233 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-tp2ll" Jul 14 11:48:06.681131 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.681097 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd"] Jul 14 11:48:06.683361 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.683342 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.687879 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.687835 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8qpv8\"" Jul 14 11:48:06.687879 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.687840 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 11:48:06.688064 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.687993 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 14 11:48:06.688203 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.688138 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 11:48:06.696354 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.696331 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd"] Jul 14 11:48:06.748045 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.748025 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.748193 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.748066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.748193 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.748109 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.748193 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.748130 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d57667c5-b5ca-478b-af9c-37ac0365af12-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.748193 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.748164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb22t\" (UniqueName: \"kubernetes.io/projected/d57667c5-b5ca-478b-af9c-37ac0365af12-kube-api-access-rb22t\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.748362 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.748217 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.848920 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.848882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.848920 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.848923 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.849140 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.848941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.849140 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.848957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d57667c5-b5ca-478b-af9c-37ac0365af12-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.849140 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.848993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb22t\" (UniqueName: \"kubernetes.io/projected/d57667c5-b5ca-478b-af9c-37ac0365af12-kube-api-access-rb22t\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.849140 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.849018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.849377 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.849359 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.849435 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.849359 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.849435 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.849424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.851223 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.851206 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d57667c5-b5ca-478b-af9c-37ac0365af12-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.851483 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.851468 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d57667c5-b5ca-478b-af9c-37ac0365af12-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.860777 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.860757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb22t\" (UniqueName: \"kubernetes.io/projected/d57667c5-b5ca-478b-af9c-37ac0365af12-kube-api-access-rb22t\") pod \"e2e-distinct-simulated-kserve-59975d69f9-chmdd\" (UID: \"d57667c5-b5ca-478b-af9c-37ac0365af12\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:06.994031 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:06.994010 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:07.120996 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:07.120959 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd"] Jul 14 11:48:07.121788 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:48:07.121762 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57667c5_b5ca_478b_af9c_37ac0365af12.slice/crio-bb26624c769ef43134c110600ce85eeb7359d952185fd0f23a119113a995b201 WatchSource:0}: Error finding container bb26624c769ef43134c110600ce85eeb7359d952185fd0f23a119113a995b201: Status 404 returned error can't find the container with id bb26624c769ef43134c110600ce85eeb7359d952185fd0f23a119113a995b201 Jul 14 11:48:07.123056 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:07.123037 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 11:48:07.414202 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:07.414127 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" event={"ID":"d57667c5-b5ca-478b-af9c-37ac0365af12","Type":"ContainerStarted","Data":"bb26624c769ef43134c110600ce85eeb7359d952185fd0f23a119113a995b201"} Jul 14 11:48:08.419997 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:08.419957 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" event={"ID":"d57667c5-b5ca-478b-af9c-37ac0365af12","Type":"ContainerStarted","Data":"e4769a7bac43df23089dcee3dbadaf967376973055737b2fe44e519d8f99fb6e"} Jul 14 11:48:08.443589 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:08.443502 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" podStartSLOduration=1.382865461 podStartE2EDuration="2.443487626s" totalImagesPullingTime="1.060622165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:48:08.441528708 +0000 UTC m=+621.517383309" watchObservedRunningTime="2026-07-14 11:48:08.443487626 +0000 UTC m=+621.519342264" Jul 14 11:48:16.994448 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:16.994343 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:16.994448 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:16.994409 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:17.006959 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:17.006932 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:17.483838 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:17.483803 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-chmdd" Jul 14 11:48:21.368217 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.368179 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm"] Jul 14 11:48:21.371533 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.371513 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.374665 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.374646 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 14 11:48:21.386906 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.386881 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm"] Jul 14 11:48:21.462152 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.462127 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6672e86-5394-4557-9520-e179140a3e4c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.462258 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.462164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.462305 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.462247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.462361 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.462309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mq7c\" (UniqueName: \"kubernetes.io/projected/c6672e86-5394-4557-9520-e179140a3e4c-kube-api-access-7mq7c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.462439 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.462420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.462505 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.462455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.563056 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.563032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6672e86-5394-4557-9520-e179140a3e4c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.563189 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.563061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.563189 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.563099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.563189 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.563128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mq7c\" (UniqueName: \"kubernetes.io/projected/c6672e86-5394-4557-9520-e179140a3e4c-kube-api-access-7mq7c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.563189 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.563159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.563189 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.563177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.563530 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.563502 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.563634 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.563549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.563677 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.563656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.565268 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.565248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6672e86-5394-4557-9520-e179140a3e4c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.565426 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.565408 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6672e86-5394-4557-9520-e179140a3e4c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.580006 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.579986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mq7c\" (UniqueName: \"kubernetes.io/projected/c6672e86-5394-4557-9520-e179140a3e4c-kube-api-access-7mq7c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm\" (UID: \"c6672e86-5394-4557-9520-e179140a3e4c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.681379 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.681318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:21.807109 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:21.807073 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm"] Jul 14 11:48:21.807707 ip-10-0-132-116 kubenswrapper[2562]: W0714 11:48:21.807678 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6672e86_5394_4557_9520_e179140a3e4c.slice/crio-c9b218f97387a77e0e0e453c5cea30d7758e8c9d4a595c7311601d6edb89962d WatchSource:0}: Error finding container c9b218f97387a77e0e0e453c5cea30d7758e8c9d4a595c7311601d6edb89962d: Status 404 returned error can't find the container with id c9b218f97387a77e0e0e453c5cea30d7758e8c9d4a595c7311601d6edb89962d Jul 14 11:48:22.493117 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:22.493062 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" event={"ID":"c6672e86-5394-4557-9520-e179140a3e4c","Type":"ContainerStarted","Data":"cf6f1d9b8c728518abf87e977d2b2edc85a01dc52c5f30b2dee7a18ef19f6b29"} Jul 14 11:48:22.493565 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:22.493129 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" event={"ID":"c6672e86-5394-4557-9520-e179140a3e4c","Type":"ContainerStarted","Data":"c9b218f97387a77e0e0e453c5cea30d7758e8c9d4a595c7311601d6edb89962d"} Jul 14 11:48:22.513667 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:22.513613 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" podStartSLOduration=1.321259467 podStartE2EDuration="1.513598264s" totalImagesPullingTime="192.338797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 11:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 11:48:22.512420372 +0000 UTC m=+635.588274964" watchObservedRunningTime="2026-07-14 11:48:22.513598264 +0000 UTC m=+635.589452851" Jul 14 11:48:31.682428 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:31.682386 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:31.682875 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:31.682443 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:31.694390 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:31.694364 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:48:32.556579 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:48:32.556548 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm" Jul 14 11:49:49.804833 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:49.804747 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c4d6c7474-x6rcn"] Jul 14 11:49:49.805338 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:49.804971 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" podUID="21dbc61d-fbfe-41ae-8232-b9278badad92" containerName="authorino" containerID="cri-o://9a674f7d3e51385c5c4d8af8d702e41e9f86e1821fe0f6db171187cd1435ded0" gracePeriod=30 Jul 14 11:49:49.949494 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:49.949462 2562 generic.go:356] "Generic (PLEG): container finished" podID="21dbc61d-fbfe-41ae-8232-b9278badad92" containerID="9a674f7d3e51385c5c4d8af8d702e41e9f86e1821fe0f6db171187cd1435ded0" exitCode=0 Jul 14 11:49:49.949607 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:49.949531 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" event={"ID":"21dbc61d-fbfe-41ae-8232-b9278badad92","Type":"ContainerDied","Data":"9a674f7d3e51385c5c4d8af8d702e41e9f86e1821fe0f6db171187cd1435ded0"} Jul 14 11:49:50.056746 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.056691 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:49:50.140004 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.139970 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21dbc61d-fbfe-41ae-8232-b9278badad92-tls-cert\") pod \"21dbc61d-fbfe-41ae-8232-b9278badad92\" (UID: \"21dbc61d-fbfe-41ae-8232-b9278badad92\") " Jul 14 11:49:50.140176 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.140121 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlsgr\" (UniqueName: \"kubernetes.io/projected/21dbc61d-fbfe-41ae-8232-b9278badad92-kube-api-access-zlsgr\") pod \"21dbc61d-fbfe-41ae-8232-b9278badad92\" (UID: \"21dbc61d-fbfe-41ae-8232-b9278badad92\") " Jul 14 11:49:50.142014 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.141988 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dbc61d-fbfe-41ae-8232-b9278badad92-kube-api-access-zlsgr" (OuterVolumeSpecName: "kube-api-access-zlsgr") pod "21dbc61d-fbfe-41ae-8232-b9278badad92" (UID: "21dbc61d-fbfe-41ae-8232-b9278badad92"). InnerVolumeSpecName "kube-api-access-zlsgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 11:49:50.149848 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.149816 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbc61d-fbfe-41ae-8232-b9278badad92-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "21dbc61d-fbfe-41ae-8232-b9278badad92" (UID: "21dbc61d-fbfe-41ae-8232-b9278badad92"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 11:49:50.241762 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.241735 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlsgr\" (UniqueName: \"kubernetes.io/projected/21dbc61d-fbfe-41ae-8232-b9278badad92-kube-api-access-zlsgr\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:49:50.241762 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.241760 2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21dbc61d-fbfe-41ae-8232-b9278badad92-tls-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 11:49:50.956596 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.956567 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" Jul 14 11:49:50.957005 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.956613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c4d6c7474-x6rcn" event={"ID":"21dbc61d-fbfe-41ae-8232-b9278badad92","Type":"ContainerDied","Data":"f41837cbe6a6d6739bdc542c0db1d448468a8eae9c781c2e763b079a071aa31e"} Jul 14 11:49:50.957005 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.956661 2562 scope.go:117] "RemoveContainer" containerID="9a674f7d3e51385c5c4d8af8d702e41e9f86e1821fe0f6db171187cd1435ded0" Jul 14 11:49:50.989608 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.989581 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c4d6c7474-x6rcn"] Jul 14 11:49:50.994103 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:50.994064 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c4d6c7474-x6rcn"] Jul 14 11:49:51.479377 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:49:51.479347 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21dbc61d-fbfe-41ae-8232-b9278badad92" path="/var/lib/kubelet/pods/21dbc61d-fbfe-41ae-8232-b9278badad92/volumes" Jul 14 11:52:47.357558 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:52:47.357532 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:52:47.358408 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:52:47.358387 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:52:47.361175 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:52:47.361159 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:52:47.362019 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:52:47.362001 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:57:47.389618 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:57:47.389590 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:57:47.391071 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:57:47.391050 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 11:57:47.392889 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:57:47.392872 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 11:57:47.394449 ip-10-0-132-116 kubenswrapper[2562]: I0714 11:57:47.394431 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:00:00.166024 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.165976 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz"] Jul 14 12:00:00.166697 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.166533 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21dbc61d-fbfe-41ae-8232-b9278badad92" containerName="authorino" Jul 14 12:00:00.166697 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.166555 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="21dbc61d-fbfe-41ae-8232-b9278badad92" containerName="authorino" Jul 14 12:00:00.166697 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.166655 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="21dbc61d-fbfe-41ae-8232-b9278badad92" containerName="authorino" Jul 14 12:00:00.169722 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.169697 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz" Jul 14 12:00:00.172918 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.172898 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:00:00.179422 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.179404 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz" Jul 14 12:00:00.197349 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.197323 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz"] Jul 14 12:00:00.314509 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.314481 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz"] Jul 14 12:00:00.315860 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:00:00.315835 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc0ee43_8711_4276_9c3a_e950b55a1e6e.slice/crio-3faf0c60e058f3248311107190edbcd8f289b89ac2aaa9d7d7c9190dd2963fe2 WatchSource:0}: Error finding container 3faf0c60e058f3248311107190edbcd8f289b89ac2aaa9d7d7c9190dd2963fe2: Status 404 returned error can't find the container with id 3faf0c60e058f3248311107190edbcd8f289b89ac2aaa9d7d7c9190dd2963fe2 Jul 14 12:00:00.317712 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:00.317692 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 12:00:01.060791 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:01.060747 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz" event={"ID":"ecc0ee43-8711-4276-9c3a-e950b55a1e6e","Type":"ContainerStarted","Data":"3faf0c60e058f3248311107190edbcd8f289b89ac2aaa9d7d7c9190dd2963fe2"} Jul 14 12:00:23.179332 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:23.179298 2562 generic.go:356] "Generic (PLEG): container finished" podID="ecc0ee43-8711-4276-9c3a-e950b55a1e6e" containerID="9dcab9094593bf9f334d38997b87516364e61c3fb842ab92c17892fc52c76335" exitCode=0 Jul 14 12:00:23.179702 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:23.179384 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz" event={"ID":"ecc0ee43-8711-4276-9c3a-e950b55a1e6e","Type":"ContainerDied","Data":"9dcab9094593bf9f334d38997b87516364e61c3fb842ab92c17892fc52c76335"} Jul 14 12:00:24.316786 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:24.316760 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz" Jul 14 12:00:25.191454 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:25.191421 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz" Jul 14 12:00:25.191454 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:25.191441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733840-ftwtz" event={"ID":"ecc0ee43-8711-4276-9c3a-e950b55a1e6e","Type":"ContainerDied","Data":"3faf0c60e058f3248311107190edbcd8f289b89ac2aaa9d7d7c9190dd2963fe2"} Jul 14 12:00:25.191670 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:00:25.191487 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3faf0c60e058f3248311107190edbcd8f289b89ac2aaa9d7d7c9190dd2963fe2" Jul 14 12:02:47.421992 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:02:47.421961 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:02:47.424650 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:02:47.424631 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:02:47.425611 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:02:47.425592 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:02:47.427972 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:02:47.427954 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:03:21.729250 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.729221 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf"] Jul 14 12:03:21.729704 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.729530 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc0ee43-8711-4276-9c3a-e950b55a1e6e" containerName="cleanup" Jul 14 12:03:21.729704 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.729541 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc0ee43-8711-4276-9c3a-e950b55a1e6e" containerName="cleanup" Jul 14 12:03:21.729704 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.729613 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc0ee43-8711-4276-9c3a-e950b55a1e6e" containerName="cleanup" Jul 14 12:03:21.732467 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.732448 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.735391 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.735373 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ab110be5-openshift-default-dockercfg-4lj69\"" Jul 14 12:03:21.755164 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.755144 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf"] Jul 14 12:03:21.789874 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.789850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-certs\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.789949 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.789881 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-istio-token\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.789949 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.789913 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-socket\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.790033 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.789980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416388ea-486b-4330-ad27-971fc228a0f5-istio-podinfo\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.790033 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.790000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416388ea-486b-4330-ad27-971fc228a0f5-istiod-ca-cert\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.790134 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.790037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-data\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.790134 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.790069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6658w\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-kube-api-access-6658w\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.790227 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.790135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-credential-socket\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.790227 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.790159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-envoy\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891527 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-certs\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891651 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-istio-token\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891651 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-socket\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891651 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416388ea-486b-4330-ad27-971fc228a0f5-istio-podinfo\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891651 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891608 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416388ea-486b-4330-ad27-971fc228a0f5-istiod-ca-cert\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891651 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-data\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891651 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6658w\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-kube-api-access-6658w\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891932 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891662 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-credential-socket\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891932 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891684 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-envoy\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.891932 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891907 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-certs\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.892114 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.891970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-socket\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.892114 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.892035 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-data\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.892235 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.892155 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-credential-socket\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.892600 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.892577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416388ea-486b-4330-ad27-971fc228a0f5-istiod-ca-cert\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.893792 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.893767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-envoy\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.893972 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.893955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416388ea-486b-4330-ad27-971fc228a0f5-istio-podinfo\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.900943 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.900919 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-istio-token\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:21.901275 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:21.901253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6658w\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-kube-api-access-6658w\") pod \"e2e-ait-ab110be5-openshift-default-766758869c-v49jf\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:22.043360 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:22.043300 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:22.173902 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:22.173877 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf"] Jul 14 12:03:22.175727 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:03:22.175698 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416388ea_486b_4330_ad27_971fc228a0f5.slice/crio-ff8038a1a779abbf5f2df79d64f3673d909a35d5142a5f59fcac36662db88961 WatchSource:0}: Error finding container ff8038a1a779abbf5f2df79d64f3673d909a35d5142a5f59fcac36662db88961: Status 404 returned error can't find the container with id ff8038a1a779abbf5f2df79d64f3673d909a35d5142a5f59fcac36662db88961 Jul 14 12:03:22.177826 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:22.177784 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:03:22.177897 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:22.177878 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:03:22.177950 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:22.177935 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:03:23.080672 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:23.080633 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" event={"ID":"416388ea-486b-4330-ad27-971fc228a0f5","Type":"ContainerStarted","Data":"cc9667c91eae4968077b997406f0778fc899a4512df5508334afbffd55ad53c5"} Jul 14 12:03:23.081142 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:23.080693 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" event={"ID":"416388ea-486b-4330-ad27-971fc228a0f5","Type":"ContainerStarted","Data":"ff8038a1a779abbf5f2df79d64f3673d909a35d5142a5f59fcac36662db88961"} Jul 14 12:03:23.111894 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:23.111843 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" podStartSLOduration=2.111824742 podStartE2EDuration="2.111824742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:03:23.109819989 +0000 UTC m=+1536.185674572" watchObservedRunningTime="2026-07-14 12:03:23.111824742 +0000 UTC m=+1536.187679331" Jul 14 12:03:24.044329 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:24.044297 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:24.048946 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:24.048924 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:24.086384 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:24.086358 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:24.087102 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:24.087060 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:27.980907 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:27.980870 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b"] Jul 14 12:03:27.984395 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:27.984374 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" Jul 14 12:03:27.987550 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:27.987527 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:03:27.987667 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:27.987600 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:03:27.994412 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:27.994386 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b"] Jul 14 12:03:28.041643 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:28.041616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60f66f8-70f2-4af9-b98e-b6927f024904-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b\" (UID: \"f60f66f8-70f2-4af9-b98e-b6927f024904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" Jul 14 12:03:28.142046 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:28.142015 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60f66f8-70f2-4af9-b98e-b6927f024904-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b\" (UID: \"f60f66f8-70f2-4af9-b98e-b6927f024904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" Jul 14 12:03:28.142648 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:28.142629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60f66f8-70f2-4af9-b98e-b6927f024904-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b\" (UID: \"f60f66f8-70f2-4af9-b98e-b6927f024904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" Jul 14 12:03:28.295729 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:28.295649 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" Jul 14 12:03:28.429498 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:28.429469 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b"] Jul 14 12:03:28.430820 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:03:28.430796 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60f66f8_70f2_4af9_b98e_b6927f024904.slice/crio-5560f83a939b78e31d6b8eb899b6dd680a094e80306db73c77c1cf1f3e8fca3a WatchSource:0}: Error finding container 5560f83a939b78e31d6b8eb899b6dd680a094e80306db73c77c1cf1f3e8fca3a: Status 404 returned error can't find the container with id 5560f83a939b78e31d6b8eb899b6dd680a094e80306db73c77c1cf1f3e8fca3a Jul 14 12:03:29.118677 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:29.118646 2562 generic.go:356] "Generic (PLEG): container finished" podID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerID="717d1b3877ea9117197d061e6b2797506d07a1d370009519d0fb02635701e10c" exitCode=7 Jul 14 12:03:29.118998 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:29.118702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" event={"ID":"f60f66f8-70f2-4af9-b98e-b6927f024904","Type":"ContainerDied","Data":"717d1b3877ea9117197d061e6b2797506d07a1d370009519d0fb02635701e10c"} Jul 14 12:03:29.118998 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:29.118765 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" event={"ID":"f60f66f8-70f2-4af9-b98e-b6927f024904","Type":"ContainerStarted","Data":"5560f83a939b78e31d6b8eb899b6dd680a094e80306db73c77c1cf1f3e8fca3a"} Jul 14 12:03:29.118998 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:29.118925 2562 scope.go:117] "RemoveContainer" containerID="717d1b3877ea9117197d061e6b2797506d07a1d370009519d0fb02635701e10c" Jul 14 12:03:30.126163 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:30.126052 2562 generic.go:356] "Generic (PLEG): container finished" podID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerID="07418315c8d6bdeb3f58a80e4535d9622fbc93dc92883c037fd114e81beadfba" exitCode=7 Jul 14 12:03:30.126163 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:30.126142 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" event={"ID":"f60f66f8-70f2-4af9-b98e-b6927f024904","Type":"ContainerDied","Data":"07418315c8d6bdeb3f58a80e4535d9622fbc93dc92883c037fd114e81beadfba"} Jul 14 12:03:30.126634 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:30.126213 2562 scope.go:117] "RemoveContainer" containerID="717d1b3877ea9117197d061e6b2797506d07a1d370009519d0fb02635701e10c" Jul 14 12:03:30.126634 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:30.126369 2562 scope.go:117] "RemoveContainer" containerID="07418315c8d6bdeb3f58a80e4535d9622fbc93dc92883c037fd114e81beadfba" Jul 14 12:03:30.126634 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:03:30.126621 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b_odh-ai-gateway-infra(f60f66f8-70f2-4af9-b98e-b6927f024904)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" Jul 14 12:03:31.133434 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:31.133405 2562 scope.go:117] "RemoveContainer" containerID="07418315c8d6bdeb3f58a80e4535d9622fbc93dc92883c037fd114e81beadfba" Jul 14 12:03:31.133803 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:03:31.133569 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b_odh-ai-gateway-infra(f60f66f8-70f2-4af9-b98e-b6927f024904)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" Jul 14 12:03:39.260101 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:39.259716 2562 patch_prober.go:28] interesting pod/e2e-ait-ab110be5-openshift-default-766758869c-v49jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:45052->10.133.0.42:15021: read: connection reset by peer" start-of-body= Jul 14 12:03:39.260101 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:39.259779 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:45052->10.133.0.42:15021: read: connection reset by peer" Jul 14 12:03:40.192414 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:40.192377 2562 generic.go:356] "Generic (PLEG): container finished" podID="416388ea-486b-4330-ad27-971fc228a0f5" containerID="cc9667c91eae4968077b997406f0778fc899a4512df5508334afbffd55ad53c5" exitCode=0 Jul 14 12:03:40.192612 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:40.192456 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" event={"ID":"416388ea-486b-4330-ad27-971fc228a0f5","Type":"ContainerDied","Data":"cc9667c91eae4968077b997406f0778fc899a4512df5508334afbffd55ad53c5"} Jul 14 12:03:40.192907 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:40.192889 2562 scope.go:117] "RemoveContainer" containerID="cc9667c91eae4968077b997406f0778fc899a4512df5508334afbffd55ad53c5" Jul 14 12:03:40.193866 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:40.193837 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:03:40.193953 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:40.193921 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:03:40.194011 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:40.193971 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:03:41.200605 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:41.200574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" event={"ID":"416388ea-486b-4330-ad27-971fc228a0f5","Type":"ContainerStarted","Data":"4b0ff83c5e701d543d652a504f60e379f7381da3cb7d894065ebb9a0f1d36061"} Jul 14 12:03:42.044155 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:42.044116 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:42.044473 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:42.044437 2562 patch_prober.go:28] interesting pod/e2e-ait-ab110be5-openshift-default-766758869c-v49jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 14 12:03:42.044577 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:42.044491 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 14 12:03:43.043918 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:43.043882 2562 patch_prober.go:28] interesting pod/e2e-ait-ab110be5-openshift-default-766758869c-v49jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 14 12:03:43.044544 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:43.044514 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 14 12:03:43.473722 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:43.473688 2562 scope.go:117] "RemoveContainer" containerID="07418315c8d6bdeb3f58a80e4535d9622fbc93dc92883c037fd114e81beadfba" Jul 14 12:03:44.044234 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:44.044132 2562 patch_prober.go:28] interesting pod/e2e-ait-ab110be5-openshift-default-766758869c-v49jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 14 12:03:44.044234 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:44.044200 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 14 12:03:44.217255 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:44.217225 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" event={"ID":"f60f66f8-70f2-4af9-b98e-b6927f024904","Type":"ContainerStarted","Data":"bfea0c9a7f85f741c33762b4dbbd25dd28b02f2ad02198df1496712c4a564df5"} Jul 14 12:03:45.043929 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:45.043883 2562 patch_prober.go:28] interesting pod/e2e-ait-ab110be5-openshift-default-766758869c-v49jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 14 12:03:45.044130 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:45.043955 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 14 12:03:45.224579 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:45.224543 2562 generic.go:356] "Generic (PLEG): container finished" podID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerID="bfea0c9a7f85f741c33762b4dbbd25dd28b02f2ad02198df1496712c4a564df5" exitCode=0 Jul 14 12:03:45.225000 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:45.224594 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" event={"ID":"f60f66f8-70f2-4af9-b98e-b6927f024904","Type":"ContainerDied","Data":"bfea0c9a7f85f741c33762b4dbbd25dd28b02f2ad02198df1496712c4a564df5"} Jul 14 12:03:45.225000 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:45.224643 2562 scope.go:117] "RemoveContainer" containerID="07418315c8d6bdeb3f58a80e4535d9622fbc93dc92883c037fd114e81beadfba" Jul 14 12:03:45.260724 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:45.260690 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b"] Jul 14 12:03:46.047152 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:46.047121 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:46.047378 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:46.047360 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:46.048108 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:46.048066 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:03:46.361296 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:46.361276 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" Jul 14 12:03:46.506859 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:46.506820 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60f66f8-70f2-4af9-b98e-b6927f024904-maas-api-service-ca\") pod \"f60f66f8-70f2-4af9-b98e-b6927f024904\" (UID: \"f60f66f8-70f2-4af9-b98e-b6927f024904\") " Jul 14 12:03:46.507199 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:46.507174 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60f66f8-70f2-4af9-b98e-b6927f024904-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f60f66f8-70f2-4af9-b98e-b6927f024904" (UID: "f60f66f8-70f2-4af9-b98e-b6927f024904"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:03:46.608031 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:46.607820 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60f66f8-70f2-4af9-b98e-b6927f024904-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:03:47.237750 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:47.237715 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" Jul 14 12:03:47.237986 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:47.237744 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b" event={"ID":"f60f66f8-70f2-4af9-b98e-b6927f024904","Type":"ContainerDied","Data":"5560f83a939b78e31d6b8eb899b6dd680a094e80306db73c77c1cf1f3e8fca3a"} Jul 14 12:03:47.237986 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:47.237814 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5560f83a939b78e31d6b8eb899b6dd680a094e80306db73c77c1cf1f3e8fca3a" Jul 14 12:03:47.278986 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:47.278953 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b"] Jul 14 12:03:47.285461 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:47.285432 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-mrq6b"] Jul 14 12:03:47.478972 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:03:47.478935 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" path="/var/lib/kubelet/pods/f60f66f8-70f2-4af9-b98e-b6927f024904/volumes" Jul 14 12:04:18.029563 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.029531 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7"] Jul 14 12:04:18.030028 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.029862 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerName="revoke-keys" Jul 14 12:04:18.030028 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.029872 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerName="revoke-keys" Jul 14 12:04:18.030028 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.029897 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerName="revoke-keys" Jul 14 12:04:18.030028 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.029902 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerName="revoke-keys" Jul 14 12:04:18.030028 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.029953 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerName="revoke-keys" Jul 14 12:04:18.030028 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.029963 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerName="revoke-keys" Jul 14 12:04:18.030028 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.029969 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerName="revoke-keys" Jul 14 12:04:18.032044 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.032027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" Jul 14 12:04:18.035214 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.035195 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:04:18.035317 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.035287 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:04:18.042037 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.042004 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7"] Jul 14 12:04:18.081765 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.081738 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ab358c-5910-40c3-95b4-552358130286-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7\" (UID: \"93ab358c-5910-40c3-95b4-552358130286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" Jul 14 12:04:18.182213 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.182185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ab358c-5910-40c3-95b4-552358130286-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7\" (UID: \"93ab358c-5910-40c3-95b4-552358130286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" Jul 14 12:04:18.182765 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.182748 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ab358c-5910-40c3-95b4-552358130286-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7\" (UID: \"93ab358c-5910-40c3-95b4-552358130286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" Jul 14 12:04:18.343653 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.343587 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" Jul 14 12:04:18.474843 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:18.474820 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7"] Jul 14 12:04:18.476777 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:04:18.476742 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ab358c_5910_40c3_95b4_552358130286.slice/crio-73ec16dd2a6fde998003941026be9c76554f62753fcb0379def1f1fa6f5d2d8d WatchSource:0}: Error finding container 73ec16dd2a6fde998003941026be9c76554f62753fcb0379def1f1fa6f5d2d8d: Status 404 returned error can't find the container with id 73ec16dd2a6fde998003941026be9c76554f62753fcb0379def1f1fa6f5d2d8d Jul 14 12:04:19.413880 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:19.413788 2562 generic.go:356] "Generic (PLEG): container finished" podID="93ab358c-5910-40c3-95b4-552358130286" containerID="680ffddcee8cbe2f7a7fcd8880fe356a7d0b7aef040cdb2236e8963e8e62887a" exitCode=0 Jul 14 12:04:19.413880 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:19.413825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" event={"ID":"93ab358c-5910-40c3-95b4-552358130286","Type":"ContainerDied","Data":"680ffddcee8cbe2f7a7fcd8880fe356a7d0b7aef040cdb2236e8963e8e62887a"} Jul 14 12:04:19.413880 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:19.413864 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" event={"ID":"93ab358c-5910-40c3-95b4-552358130286","Type":"ContainerStarted","Data":"73ec16dd2a6fde998003941026be9c76554f62753fcb0379def1f1fa6f5d2d8d"} Jul 14 12:04:20.545903 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:20.545873 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" Jul 14 12:04:20.604026 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:20.603997 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ab358c-5910-40c3-95b4-552358130286-maas-api-service-ca\") pod \"93ab358c-5910-40c3-95b4-552358130286\" (UID: \"93ab358c-5910-40c3-95b4-552358130286\") " Jul 14 12:04:20.604368 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:20.604347 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ab358c-5910-40c3-95b4-552358130286-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93ab358c-5910-40c3-95b4-552358130286" (UID: "93ab358c-5910-40c3-95b4-552358130286"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:04:20.705296 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:20.705226 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ab358c-5910-40c3-95b4-552358130286-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:21.425592 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:21.425560 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" Jul 14 12:04:21.425767 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:21.425594 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7" event={"ID":"93ab358c-5910-40c3-95b4-552358130286","Type":"ContainerDied","Data":"73ec16dd2a6fde998003941026be9c76554f62753fcb0379def1f1fa6f5d2d8d"} Jul 14 12:04:21.425767 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:21.425637 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ec16dd2a6fde998003941026be9c76554f62753fcb0379def1f1fa6f5d2d8d" Jul 14 12:04:43.370657 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:43.370575 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf"] Jul 14 12:04:43.371455 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:43.370797 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" containerID="cri-o://4b0ff83c5e701d543d652a504f60e379f7381da3cb7d894065ebb9a0f1d36061" gracePeriod=30 Jul 14 12:04:46.049753 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:46.049716 2562 patch_prober.go:28] interesting pod/e2e-ait-ab110be5-openshift-default-766758869c-v49jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 12:04:46.050164 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:46.049779 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 12:04:48.569073 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.569039 2562 generic.go:356] "Generic (PLEG): container finished" podID="416388ea-486b-4330-ad27-971fc228a0f5" containerID="4b0ff83c5e701d543d652a504f60e379f7381da3cb7d894065ebb9a0f1d36061" exitCode=0 Jul 14 12:04:48.569410 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.569108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" event={"ID":"416388ea-486b-4330-ad27-971fc228a0f5","Type":"ContainerDied","Data":"4b0ff83c5e701d543d652a504f60e379f7381da3cb7d894065ebb9a0f1d36061"} Jul 14 12:04:48.569410 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.569170 2562 scope.go:117] "RemoveContainer" containerID="cc9667c91eae4968077b997406f0778fc899a4512df5508334afbffd55ad53c5" Jul 14 12:04:48.620471 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.620453 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:04:48.723885 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.723860 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-data\") pod \"416388ea-486b-4330-ad27-971fc228a0f5\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " Jul 14 12:04:48.724026 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.723908 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-envoy\") pod \"416388ea-486b-4330-ad27-971fc228a0f5\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " Jul 14 12:04:48.724026 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.723946 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-istio-token\") pod \"416388ea-486b-4330-ad27-971fc228a0f5\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " Jul 14 12:04:48.724026 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.723968 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-socket\") pod \"416388ea-486b-4330-ad27-971fc228a0f5\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " Jul 14 12:04:48.724026 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724016 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416388ea-486b-4330-ad27-971fc228a0f5-istiod-ca-cert\") pod \"416388ea-486b-4330-ad27-971fc228a0f5\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " Jul 14 12:04:48.724262 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724039 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-credential-socket\") pod \"416388ea-486b-4330-ad27-971fc228a0f5\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " Jul 14 12:04:48.724262 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724067 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-certs\") pod \"416388ea-486b-4330-ad27-971fc228a0f5\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " Jul 14 12:04:48.724262 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724110 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6658w\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-kube-api-access-6658w\") pod \"416388ea-486b-4330-ad27-971fc228a0f5\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " Jul 14 12:04:48.724262 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724137 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416388ea-486b-4330-ad27-971fc228a0f5-istio-podinfo\") pod \"416388ea-486b-4330-ad27-971fc228a0f5\" (UID: \"416388ea-486b-4330-ad27-971fc228a0f5\") " Jul 14 12:04:48.724461 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724366 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "416388ea-486b-4330-ad27-971fc228a0f5" (UID: "416388ea-486b-4330-ad27-971fc228a0f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:04:48.724526 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724484 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "416388ea-486b-4330-ad27-971fc228a0f5" (UID: "416388ea-486b-4330-ad27-971fc228a0f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:04:48.724526 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724504 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416388ea-486b-4330-ad27-971fc228a0f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "416388ea-486b-4330-ad27-971fc228a0f5" (UID: "416388ea-486b-4330-ad27-971fc228a0f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:04:48.724526 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724503 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "416388ea-486b-4330-ad27-971fc228a0f5" (UID: "416388ea-486b-4330-ad27-971fc228a0f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:04:48.724705 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.724517 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:48.725160 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.725131 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "416388ea-486b-4330-ad27-971fc228a0f5" (UID: "416388ea-486b-4330-ad27-971fc228a0f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:04:48.726467 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.726440 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "416388ea-486b-4330-ad27-971fc228a0f5" (UID: "416388ea-486b-4330-ad27-971fc228a0f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:04:48.726630 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.726597 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-kube-api-access-6658w" (OuterVolumeSpecName: "kube-api-access-6658w") pod "416388ea-486b-4330-ad27-971fc228a0f5" (UID: "416388ea-486b-4330-ad27-971fc228a0f5"). InnerVolumeSpecName "kube-api-access-6658w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:04:48.726630 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.726603 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/416388ea-486b-4330-ad27-971fc228a0f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "416388ea-486b-4330-ad27-971fc228a0f5" (UID: "416388ea-486b-4330-ad27-971fc228a0f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:04:48.726756 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.726654 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "416388ea-486b-4330-ad27-971fc228a0f5" (UID: "416388ea-486b-4330-ad27-971fc228a0f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:04:48.825289 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.825266 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:48.825289 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.825286 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:48.825482 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.825294 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416388ea-486b-4330-ad27-971fc228a0f5-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:48.825482 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.825303 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:48.825482 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.825313 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:48.825482 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.825321 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6658w\" (UniqueName: \"kubernetes.io/projected/416388ea-486b-4330-ad27-971fc228a0f5-kube-api-access-6658w\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:48.825482 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.825329 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416388ea-486b-4330-ad27-971fc228a0f5-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:48.825482 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:48.825337 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416388ea-486b-4330-ad27-971fc228a0f5-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:04:49.575567 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.575534 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" event={"ID":"416388ea-486b-4330-ad27-971fc228a0f5","Type":"ContainerDied","Data":"ff8038a1a779abbf5f2df79d64f3673d909a35d5142a5f59fcac36662db88961"} Jul 14 12:04:49.575880 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.575586 2562 scope.go:117] "RemoveContainer" containerID="4b0ff83c5e701d543d652a504f60e379f7381da3cb7d894065ebb9a0f1d36061" Jul 14 12:04:49.575880 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.575549 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf" Jul 14 12:04:49.602258 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.602233 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf"] Jul 14 12:04:49.608208 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.608187 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ab110be5-openshift-default-766758869c-v49jf"] Jul 14 12:04:49.786561 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786500 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf"] Jul 14 12:04:49.786814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786801 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ab358c-5910-40c3-95b4-552358130286" containerName="revoke-keys" Jul 14 12:04:49.786867 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786814 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ab358c-5910-40c3-95b4-552358130286" containerName="revoke-keys" Jul 14 12:04:49.786867 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786823 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" Jul 14 12:04:49.786867 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786829 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" Jul 14 12:04:49.786867 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786837 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" Jul 14 12:04:49.786867 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786843 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" Jul 14 12:04:49.786867 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786857 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerName="revoke-keys" Jul 14 12:04:49.786867 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786862 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60f66f8-70f2-4af9-b98e-b6927f024904" containerName="revoke-keys" Jul 14 12:04:49.787123 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786917 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" Jul 14 12:04:49.787123 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786923 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ab358c-5910-40c3-95b4-552358130286" containerName="revoke-keys" Jul 14 12:04:49.787123 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.786931 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="416388ea-486b-4330-ad27-971fc228a0f5" containerName="istio-proxy" Jul 14 12:04:49.789543 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.789527 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" Jul 14 12:04:49.792483 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.792465 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:04:49.792570 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.792550 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:04:49.800055 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.800032 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf"] Jul 14 12:04:49.934244 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:49.934207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1b457-451f-425c-8c41-8f9a21e794ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf\" (UID: \"5df1b457-451f-425c-8c41-8f9a21e794ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" Jul 14 12:04:50.034890 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:50.034863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1b457-451f-425c-8c41-8f9a21e794ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf\" (UID: \"5df1b457-451f-425c-8c41-8f9a21e794ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" Jul 14 12:04:50.035527 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:50.035509 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1b457-451f-425c-8c41-8f9a21e794ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf\" (UID: \"5df1b457-451f-425c-8c41-8f9a21e794ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" Jul 14 12:04:50.100590 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:50.100517 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" Jul 14 12:04:50.239104 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:50.239063 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf"] Jul 14 12:04:50.240410 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:04:50.240387 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df1b457_451f_425c_8c41_8f9a21e794ab.slice/crio-f9edb037e0ac0213023a92ce14e478e0f71071f5d34c0025d578c7cb9dfeb9dc WatchSource:0}: Error finding container f9edb037e0ac0213023a92ce14e478e0f71071f5d34c0025d578c7cb9dfeb9dc: Status 404 returned error can't find the container with id f9edb037e0ac0213023a92ce14e478e0f71071f5d34c0025d578c7cb9dfeb9dc Jul 14 12:04:50.583364 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:50.583329 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" event={"ID":"5df1b457-451f-425c-8c41-8f9a21e794ab","Type":"ContainerStarted","Data":"f992e7dfaef0fe25839d8d788bbf2a847a87a89548f24605e1c1620a25635e4a"} Jul 14 12:04:50.583736 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:50.583386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" event={"ID":"5df1b457-451f-425c-8c41-8f9a21e794ab","Type":"ContainerStarted","Data":"f9edb037e0ac0213023a92ce14e478e0f71071f5d34c0025d578c7cb9dfeb9dc"} Jul 14 12:04:50.613588 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:50.613539 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" podStartSLOduration=1.613522884 podStartE2EDuration="1.613522884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:04:50.611906267 +0000 UTC m=+1623.687760855" watchObservedRunningTime="2026-07-14 12:04:50.613522884 +0000 UTC m=+1623.689377472" Jul 14 12:04:51.481202 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:51.481160 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="416388ea-486b-4330-ad27-971fc228a0f5" path="/var/lib/kubelet/pods/416388ea-486b-4330-ad27-971fc228a0f5/volumes" Jul 14 12:04:51.590153 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:51.590123 2562 generic.go:356] "Generic (PLEG): container finished" podID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerID="f992e7dfaef0fe25839d8d788bbf2a847a87a89548f24605e1c1620a25635e4a" exitCode=7 Jul 14 12:04:51.590500 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:51.590188 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" event={"ID":"5df1b457-451f-425c-8c41-8f9a21e794ab","Type":"ContainerDied","Data":"f992e7dfaef0fe25839d8d788bbf2a847a87a89548f24605e1c1620a25635e4a"} Jul 14 12:04:51.590500 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:51.590351 2562 scope.go:117] "RemoveContainer" containerID="f992e7dfaef0fe25839d8d788bbf2a847a87a89548f24605e1c1620a25635e4a" Jul 14 12:04:52.597144 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:52.597107 2562 generic.go:356] "Generic (PLEG): container finished" podID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerID="edb194df434ae5ba9972ba8e94f813915aefccefeea2c2950479019b88859364" exitCode=7 Jul 14 12:04:52.597562 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:52.597151 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" event={"ID":"5df1b457-451f-425c-8c41-8f9a21e794ab","Type":"ContainerDied","Data":"edb194df434ae5ba9972ba8e94f813915aefccefeea2c2950479019b88859364"} Jul 14 12:04:52.597562 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:52.597208 2562 scope.go:117] "RemoveContainer" containerID="f992e7dfaef0fe25839d8d788bbf2a847a87a89548f24605e1c1620a25635e4a" Jul 14 12:04:52.597562 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:52.597416 2562 scope.go:117] "RemoveContainer" containerID="edb194df434ae5ba9972ba8e94f813915aefccefeea2c2950479019b88859364" Jul 14 12:04:52.597696 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:04:52.597644 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf_odh-ai-gateway-infra(5df1b457-451f-425c-8c41-8f9a21e794ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" Jul 14 12:04:53.605711 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:04:53.605683 2562 scope.go:117] "RemoveContainer" containerID="edb194df434ae5ba9972ba8e94f813915aefccefeea2c2950479019b88859364" Jul 14 12:04:53.606190 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:04:53.606044 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf_odh-ai-gateway-infra(5df1b457-451f-425c-8c41-8f9a21e794ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" Jul 14 12:05:05.479668 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:05.479638 2562 scope.go:117] "RemoveContainer" containerID="edb194df434ae5ba9972ba8e94f813915aefccefeea2c2950479019b88859364" Jul 14 12:05:06.672037 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:06.671998 2562 generic.go:356] "Generic (PLEG): container finished" podID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerID="8a15d978d3ba9b734eb09ade6db82defb6f90975c7daebdf0d44661395b9ea54" exitCode=0 Jul 14 12:05:06.672479 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:06.672044 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" event={"ID":"5df1b457-451f-425c-8c41-8f9a21e794ab","Type":"ContainerDied","Data":"8a15d978d3ba9b734eb09ade6db82defb6f90975c7daebdf0d44661395b9ea54"} Jul 14 12:05:06.672479 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:06.672141 2562 scope.go:117] "RemoveContainer" containerID="edb194df434ae5ba9972ba8e94f813915aefccefeea2c2950479019b88859364" Jul 14 12:05:07.709143 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:07.709108 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf"] Jul 14 12:05:07.809255 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:07.809228 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" Jul 14 12:05:07.872083 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:07.872059 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1b457-451f-425c-8c41-8f9a21e794ab-maas-api-service-ca\") pod \"5df1b457-451f-425c-8c41-8f9a21e794ab\" (UID: \"5df1b457-451f-425c-8c41-8f9a21e794ab\") " Jul 14 12:05:07.872408 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:07.872387 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df1b457-451f-425c-8c41-8f9a21e794ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5df1b457-451f-425c-8c41-8f9a21e794ab" (UID: "5df1b457-451f-425c-8c41-8f9a21e794ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:05:07.973198 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:07.973176 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df1b457-451f-425c-8c41-8f9a21e794ab-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:05:08.684170 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:08.684142 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" Jul 14 12:05:08.684343 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:08.684145 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf" event={"ID":"5df1b457-451f-425c-8c41-8f9a21e794ab","Type":"ContainerDied","Data":"f9edb037e0ac0213023a92ce14e478e0f71071f5d34c0025d578c7cb9dfeb9dc"} Jul 14 12:05:08.684343 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:08.684274 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9edb037e0ac0213023a92ce14e478e0f71071f5d34c0025d578c7cb9dfeb9dc" Jul 14 12:05:08.720240 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:08.720214 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf"] Jul 14 12:05:08.727825 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:08.727794 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-f54sf"] Jul 14 12:05:09.480501 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:09.480474 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" path="/var/lib/kubelet/pods/5df1b457-451f-425c-8c41-8f9a21e794ab/volumes" Jul 14 12:05:39.830033 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.829997 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p"] Jul 14 12:05:39.830572 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.830504 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerName="revoke-keys" Jul 14 12:05:39.830572 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.830525 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerName="revoke-keys" Jul 14 12:05:39.830572 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.830546 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerName="revoke-keys" Jul 14 12:05:39.830572 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.830555 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerName="revoke-keys" Jul 14 12:05:39.830572 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.830565 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerName="revoke-keys" Jul 14 12:05:39.830572 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.830574 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerName="revoke-keys" Jul 14 12:05:39.830880 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.830675 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerName="revoke-keys" Jul 14 12:05:39.830880 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.830688 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerName="revoke-keys" Jul 14 12:05:39.833757 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.833736 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" Jul 14 12:05:39.837163 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.837130 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:05:39.837282 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.837160 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:05:39.842496 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.842467 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p"] Jul 14 12:05:39.899657 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:39.899635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aebf37-854e-49c7-b78b-6a98b0879bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p\" (UID: \"95aebf37-854e-49c7-b78b-6a98b0879bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" Jul 14 12:05:40.000323 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:40.000302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aebf37-854e-49c7-b78b-6a98b0879bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p\" (UID: \"95aebf37-854e-49c7-b78b-6a98b0879bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" Jul 14 12:05:40.000832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:40.000815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aebf37-854e-49c7-b78b-6a98b0879bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p\" (UID: \"95aebf37-854e-49c7-b78b-6a98b0879bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" Jul 14 12:05:40.145299 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:40.145248 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" Jul 14 12:05:40.272266 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:40.272241 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p"] Jul 14 12:05:40.274102 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:05:40.274055 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95aebf37_854e_49c7_b78b_6a98b0879bed.slice/crio-6cc466c4fb0e5d9b3dce77ac1959782d3275ccde3d0e5fd0f092caeafae42969 WatchSource:0}: Error finding container 6cc466c4fb0e5d9b3dce77ac1959782d3275ccde3d0e5fd0f092caeafae42969: Status 404 returned error can't find the container with id 6cc466c4fb0e5d9b3dce77ac1959782d3275ccde3d0e5fd0f092caeafae42969 Jul 14 12:05:40.845243 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:40.845209 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" event={"ID":"95aebf37-854e-49c7-b78b-6a98b0879bed","Type":"ContainerStarted","Data":"65daa2477020312e5f9d5057e150f7fc0d2fd0e47f1dc5a463d9fab4bbcc4e5a"} Jul 14 12:05:40.845585 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:40.845274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" event={"ID":"95aebf37-854e-49c7-b78b-6a98b0879bed","Type":"ContainerStarted","Data":"6cc466c4fb0e5d9b3dce77ac1959782d3275ccde3d0e5fd0f092caeafae42969"} Jul 14 12:05:40.865249 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:40.865210 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" podStartSLOduration=1.865194705 podStartE2EDuration="1.865194705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:05:40.863498582 +0000 UTC m=+1673.939353170" watchObservedRunningTime="2026-07-14 12:05:40.865194705 +0000 UTC m=+1673.941049293" Jul 14 12:05:41.851956 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:41.851918 2562 generic.go:356] "Generic (PLEG): container finished" podID="95aebf37-854e-49c7-b78b-6a98b0879bed" containerID="65daa2477020312e5f9d5057e150f7fc0d2fd0e47f1dc5a463d9fab4bbcc4e5a" exitCode=0 Jul 14 12:05:41.852339 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:41.851962 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" event={"ID":"95aebf37-854e-49c7-b78b-6a98b0879bed","Type":"ContainerDied","Data":"65daa2477020312e5f9d5057e150f7fc0d2fd0e47f1dc5a463d9fab4bbcc4e5a"} Jul 14 12:05:42.980930 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:42.980909 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" Jul 14 12:05:43.022750 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:43.022724 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aebf37-854e-49c7-b78b-6a98b0879bed-maas-api-service-ca\") pod \"95aebf37-854e-49c7-b78b-6a98b0879bed\" (UID: \"95aebf37-854e-49c7-b78b-6a98b0879bed\") " Jul 14 12:05:43.023032 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:43.023012 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95aebf37-854e-49c7-b78b-6a98b0879bed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95aebf37-854e-49c7-b78b-6a98b0879bed" (UID: "95aebf37-854e-49c7-b78b-6a98b0879bed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:05:43.123679 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:43.123625 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95aebf37-854e-49c7-b78b-6a98b0879bed-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:05:43.865136 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:43.865044 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" Jul 14 12:05:43.865136 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:43.865053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p" event={"ID":"95aebf37-854e-49c7-b78b-6a98b0879bed","Type":"ContainerDied","Data":"6cc466c4fb0e5d9b3dce77ac1959782d3275ccde3d0e5fd0f092caeafae42969"} Jul 14 12:05:43.865136 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:43.865124 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cc466c4fb0e5d9b3dce77ac1959782d3275ccde3d0e5fd0f092caeafae42969" Jul 14 12:05:56.727233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.727201 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t"] Jul 14 12:05:56.727617 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.727600 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95aebf37-854e-49c7-b78b-6a98b0879bed" containerName="revoke-keys" Jul 14 12:05:56.727617 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.727612 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="95aebf37-854e-49c7-b78b-6a98b0879bed" containerName="revoke-keys" Jul 14 12:05:56.727688 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.727674 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5df1b457-451f-425c-8c41-8f9a21e794ab" containerName="revoke-keys" Jul 14 12:05:56.727688 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.727681 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="95aebf37-854e-49c7-b78b-6a98b0879bed" containerName="revoke-keys" Jul 14 12:05:56.730726 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.730700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.733830 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.733802 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0135d283-openshift-default-dockercfg-s64v7\"" Jul 14 12:05:56.743237 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.743208 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t"] Jul 14 12:05:56.834569 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.834484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istiod-ca-cert\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.834569 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.834524 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-data\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.834569 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.834543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-credential-socket\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.834569 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.834573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-envoy\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.834832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.834626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-certs\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.834832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.834644 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgg2p\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-kube-api-access-fgg2p\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.834832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.834717 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-podinfo\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.834832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.834736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-token\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.834832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.834768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-socket\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.935971 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.935939 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istiod-ca-cert\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.935971 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.935974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-data\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.935994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-credential-socket\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-envoy\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-certs\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgg2p\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-kube-api-access-fgg2p\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-podinfo\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-token\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936217 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-socket\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936641 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-data\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936641 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-certs\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936641 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936609 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-socket\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936808 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936702 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-credential-socket\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.936808 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.936795 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istiod-ca-cert\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.938440 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.938421 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-envoy\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.938575 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.938556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-podinfo\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.945439 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.945391 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgg2p\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-kube-api-access-fgg2p\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:56.945593 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:56.945576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-token\") pod \"e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:57.044596 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:57.044563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:57.183825 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:57.183797 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t"] Jul 14 12:05:57.184863 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:05:57.184832 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19973f19_f8e9_42aa_ab5b_6da94fc5f375.slice/crio-115e6672668b39e0ec954e2d0cd6da97c36d951021ba6ece765087d754f89c8a WatchSource:0}: Error finding container 115e6672668b39e0ec954e2d0cd6da97c36d951021ba6ece765087d754f89c8a: Status 404 returned error can't find the container with id 115e6672668b39e0ec954e2d0cd6da97c36d951021ba6ece765087d754f89c8a Jul 14 12:05:57.187060 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:57.187026 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:05:57.187169 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:57.187140 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:05:57.187211 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:57.187174 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:05:57.938927 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:57.938892 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" event={"ID":"19973f19-f8e9-42aa-ab5b-6da94fc5f375","Type":"ContainerStarted","Data":"355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e"} Jul 14 12:05:57.939420 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:57.938961 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" event={"ID":"19973f19-f8e9-42aa-ab5b-6da94fc5f375","Type":"ContainerStarted","Data":"115e6672668b39e0ec954e2d0cd6da97c36d951021ba6ece765087d754f89c8a"} Jul 14 12:05:57.969200 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:57.969152 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" podStartSLOduration=1.9691367739999999 podStartE2EDuration="1.969136774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:05:57.967550298 +0000 UTC m=+1691.043404886" watchObservedRunningTime="2026-07-14 12:05:57.969136774 +0000 UTC m=+1691.044991362" Jul 14 12:05:58.044844 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:58.044812 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:58.049923 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:58.049903 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:58.944304 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:58.944276 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:05:58.945350 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:05:58.945327 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:06:02.990023 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:02.989994 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb"] Jul 14 12:06:02.993563 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:02.993548 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" Jul 14 12:06:02.996347 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:02.996323 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:06:02.996475 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:02.996365 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:06:03.002706 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:03.002680 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb"] Jul 14 12:06:03.089592 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:03.089561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0135d283-jqqgb\" (UID: \"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" Jul 14 12:06:03.190787 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:03.190754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0135d283-jqqgb\" (UID: \"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" Jul 14 12:06:03.191402 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:03.191384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0135d283-jqqgb\" (UID: \"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" Jul 14 12:06:03.305506 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:03.305435 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" Jul 14 12:06:03.437527 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:03.437500 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb"] Jul 14 12:06:03.438998 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:06:03.438975 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcf5f4e_4113_42a0_9ac1_fc37743b5ee5.slice/crio-47ad20e3e79655c12b5d06d4f0a216763822e7824abb1a0c4eab6a392dc5b856 WatchSource:0}: Error finding container 47ad20e3e79655c12b5d06d4f0a216763822e7824abb1a0c4eab6a392dc5b856: Status 404 returned error can't find the container with id 47ad20e3e79655c12b5d06d4f0a216763822e7824abb1a0c4eab6a392dc5b856 Jul 14 12:06:03.971463 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:03.971372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" event={"ID":"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5","Type":"ContainerStarted","Data":"753d6c9562b4a3cdd8703b559ab2064ca72cf59c4588869a84802dea1590a40e"} Jul 14 12:06:03.971463 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:03.971424 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" event={"ID":"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5","Type":"ContainerStarted","Data":"47ad20e3e79655c12b5d06d4f0a216763822e7824abb1a0c4eab6a392dc5b856"} Jul 14 12:06:03.995134 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:03.995066 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" podStartSLOduration=1.995050349 podStartE2EDuration="1.995050349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:06:03.992834451 +0000 UTC m=+1697.068689044" watchObservedRunningTime="2026-07-14 12:06:03.995050349 +0000 UTC m=+1697.070905013" Jul 14 12:06:04.978198 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:04.978161 2562 generic.go:356] "Generic (PLEG): container finished" podID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerID="753d6c9562b4a3cdd8703b559ab2064ca72cf59c4588869a84802dea1590a40e" exitCode=7 Jul 14 12:06:04.978413 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:04.978249 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" event={"ID":"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5","Type":"ContainerDied","Data":"753d6c9562b4a3cdd8703b559ab2064ca72cf59c4588869a84802dea1590a40e"} Jul 14 12:06:04.978501 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:04.978483 2562 scope.go:117] "RemoveContainer" containerID="753d6c9562b4a3cdd8703b559ab2064ca72cf59c4588869a84802dea1590a40e" Jul 14 12:06:05.985394 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:05.985358 2562 generic.go:356] "Generic (PLEG): container finished" podID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerID="868544446aca54702f022c3be4e31b5e70baf67bde6c992fd8df1df7a902f861" exitCode=7 Jul 14 12:06:05.985857 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:05.985404 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" event={"ID":"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5","Type":"ContainerDied","Data":"868544446aca54702f022c3be4e31b5e70baf67bde6c992fd8df1df7a902f861"} Jul 14 12:06:05.985857 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:05.985468 2562 scope.go:117] "RemoveContainer" containerID="753d6c9562b4a3cdd8703b559ab2064ca72cf59c4588869a84802dea1590a40e" Jul 14 12:06:05.985857 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:05.985679 2562 scope.go:117] "RemoveContainer" containerID="868544446aca54702f022c3be4e31b5e70baf67bde6c992fd8df1df7a902f861" Jul 14 12:06:05.986017 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:06:05.985924 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0135d283-jqqgb_odh-ai-gateway-infra(cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" Jul 14 12:06:06.992886 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:06.992852 2562 scope.go:117] "RemoveContainer" containerID="868544446aca54702f022c3be4e31b5e70baf67bde6c992fd8df1df7a902f861" Jul 14 12:06:06.993298 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:06:06.993029 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0135d283-jqqgb_odh-ai-gateway-infra(cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" Jul 14 12:06:14.987215 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:14.987176 2562 patch_prober.go:28] interesting pod/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 12:06:14.987603 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:14.987243 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 12:06:16.041879 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:16.041843 2562 generic.go:356] "Generic (PLEG): container finished" podID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerID="355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e" exitCode=0 Jul 14 12:06:16.042338 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:16.041913 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" event={"ID":"19973f19-f8e9-42aa-ab5b-6da94fc5f375","Type":"ContainerDied","Data":"355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e"} Jul 14 12:06:16.042338 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:16.042311 2562 scope.go:117] "RemoveContainer" containerID="355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e" Jul 14 12:06:16.043259 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:16.043227 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:06:16.043332 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:16.043313 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:06:16.043390 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:16.043362 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:06:17.051829 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:17.049648 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" event={"ID":"19973f19-f8e9-42aa-ab5b-6da94fc5f375","Type":"ContainerStarted","Data":"d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758"} Jul 14 12:06:18.045754 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:18.045709 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:06:18.045972 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:18.045843 2562 patch_prober.go:28] interesting pod/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 14 12:06:18.045972 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:18.045902 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 14 12:06:19.045688 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:19.045638 2562 patch_prober.go:28] interesting pod/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 14 12:06:19.046306 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:19.045712 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 14 12:06:19.473742 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:19.473716 2562 scope.go:117] "RemoveContainer" containerID="868544446aca54702f022c3be4e31b5e70baf67bde6c992fd8df1df7a902f861" Jul 14 12:06:20.046016 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:20.045513 2562 patch_prober.go:28] interesting pod/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 14 12:06:20.046016 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:20.045580 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 14 12:06:20.071568 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:20.071539 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" event={"ID":"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5","Type":"ContainerStarted","Data":"a7871537ce0927824b31fe38d65f246d0c31c80d2f9e0dd4cde739dca4e8b782"} Jul 14 12:06:21.045656 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:21.045616 2562 patch_prober.go:28] interesting pod/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 14 12:06:21.045877 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:21.045688 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 14 12:06:21.078842 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:21.078809 2562 generic.go:356] "Generic (PLEG): container finished" podID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerID="a7871537ce0927824b31fe38d65f246d0c31c80d2f9e0dd4cde739dca4e8b782" exitCode=0 Jul 14 12:06:21.079269 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:21.078884 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" event={"ID":"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5","Type":"ContainerDied","Data":"a7871537ce0927824b31fe38d65f246d0c31c80d2f9e0dd4cde739dca4e8b782"} Jul 14 12:06:21.079269 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:21.078948 2562 scope.go:117] "RemoveContainer" containerID="868544446aca54702f022c3be4e31b5e70baf67bde6c992fd8df1df7a902f861" Jul 14 12:06:21.118763 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:21.118735 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb"] Jul 14 12:06:22.048276 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:22.048250 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:06:22.048639 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:22.048620 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:06:22.049262 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:22.049246 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:06:22.248872 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:22.248847 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" Jul 14 12:06:22.270155 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:22.270131 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5-maas-api-service-ca\") pod \"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5\" (UID: \"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5\") " Jul 14 12:06:22.270473 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:22.270449 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" (UID: "cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:06:22.371371 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:22.371308 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:06:23.096007 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:23.095974 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" Jul 14 12:06:23.096237 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:23.096018 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb" event={"ID":"cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5","Type":"ContainerDied","Data":"47ad20e3e79655c12b5d06d4f0a216763822e7824abb1a0c4eab6a392dc5b856"} Jul 14 12:06:23.096237 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:23.096069 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47ad20e3e79655c12b5d06d4f0a216763822e7824abb1a0c4eab6a392dc5b856" Jul 14 12:06:23.138962 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:23.138932 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb"] Jul 14 12:06:23.149959 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:23.149938 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-jqqgb"] Jul 14 12:06:23.478850 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:06:23.478812 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" path="/var/lib/kubelet/pods/cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5/volumes" Jul 14 12:07:03.065809 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.065775 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd"] Jul 14 12:07:03.066315 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.066131 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerName="revoke-keys" Jul 14 12:07:03.066315 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.066144 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerName="revoke-keys" Jul 14 12:07:03.066315 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.066158 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerName="revoke-keys" Jul 14 12:07:03.066315 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.066164 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerName="revoke-keys" Jul 14 12:07:03.066315 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.066230 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerName="revoke-keys" Jul 14 12:07:03.066315 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.066238 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerName="revoke-keys" Jul 14 12:07:03.069280 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.069264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" Jul 14 12:07:03.072213 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.072189 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:07:03.072341 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.072189 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:07:03.078545 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.078515 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd"] Jul 14 12:07:03.229962 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.229928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/571260b0-aa8a-41c8-860f-387924e08fc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0135d283-pfdgd\" (UID: \"571260b0-aa8a-41c8-860f-387924e08fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" Jul 14 12:07:03.330852 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.330755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/571260b0-aa8a-41c8-860f-387924e08fc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0135d283-pfdgd\" (UID: \"571260b0-aa8a-41c8-860f-387924e08fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" Jul 14 12:07:03.331447 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.331423 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/571260b0-aa8a-41c8-860f-387924e08fc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0135d283-pfdgd\" (UID: \"571260b0-aa8a-41c8-860f-387924e08fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" Jul 14 12:07:03.381297 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.381264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" Jul 14 12:07:03.520971 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:03.520945 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd"] Jul 14 12:07:03.522536 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:07:03.522509 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571260b0_aa8a_41c8_860f_387924e08fc9.slice/crio-7993c29ba49afeaed2f49c22d157dd7d324c9378072132a3a578cd9414b1327c WatchSource:0}: Error finding container 7993c29ba49afeaed2f49c22d157dd7d324c9378072132a3a578cd9414b1327c: Status 404 returned error can't find the container with id 7993c29ba49afeaed2f49c22d157dd7d324c9378072132a3a578cd9414b1327c Jul 14 12:07:04.323712 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:04.323677 2562 generic.go:356] "Generic (PLEG): container finished" podID="571260b0-aa8a-41c8-860f-387924e08fc9" containerID="4a943e3f96fcb3f146e32e29178c00ff2fd746a01c15e45ffa04c7eaf895d19c" exitCode=0 Jul 14 12:07:04.324069 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:04.323738 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" event={"ID":"571260b0-aa8a-41c8-860f-387924e08fc9","Type":"ContainerDied","Data":"4a943e3f96fcb3f146e32e29178c00ff2fd746a01c15e45ffa04c7eaf895d19c"} Jul 14 12:07:04.324069 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:04.323779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" event={"ID":"571260b0-aa8a-41c8-860f-387924e08fc9","Type":"ContainerStarted","Data":"7993c29ba49afeaed2f49c22d157dd7d324c9378072132a3a578cd9414b1327c"} Jul 14 12:07:05.466250 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:05.466224 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" Jul 14 12:07:05.550998 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:05.550965 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/571260b0-aa8a-41c8-860f-387924e08fc9-maas-api-service-ca\") pod \"571260b0-aa8a-41c8-860f-387924e08fc9\" (UID: \"571260b0-aa8a-41c8-860f-387924e08fc9\") " Jul 14 12:07:05.551361 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:05.551335 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571260b0-aa8a-41c8-860f-387924e08fc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "571260b0-aa8a-41c8-860f-387924e08fc9" (UID: "571260b0-aa8a-41c8-860f-387924e08fc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:07:05.651700 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:05.651626 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/571260b0-aa8a-41c8-860f-387924e08fc9-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:06.344590 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:06.344552 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" Jul 14 12:07:06.344774 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:06.344560 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0135d283-pfdgd" event={"ID":"571260b0-aa8a-41c8-860f-387924e08fc9","Type":"ContainerDied","Data":"7993c29ba49afeaed2f49c22d157dd7d324c9378072132a3a578cd9414b1327c"} Jul 14 12:07:06.344774 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:06.344720 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7993c29ba49afeaed2f49c22d157dd7d324c9378072132a3a578cd9414b1327c" Jul 14 12:07:29.204912 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.204871 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t"] Jul 14 12:07:29.205475 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.205157 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" containerID="cri-o://d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758" gracePeriod=30 Jul 14 12:07:29.516216 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.516178 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp"] Jul 14 12:07:29.516697 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.516671 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="571260b0-aa8a-41c8-860f-387924e08fc9" containerName="revoke-keys" Jul 14 12:07:29.516697 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.516694 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="571260b0-aa8a-41c8-860f-387924e08fc9" containerName="revoke-keys" Jul 14 12:07:29.516909 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.516725 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerName="revoke-keys" Jul 14 12:07:29.516909 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.516733 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerName="revoke-keys" Jul 14 12:07:29.516909 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.516844 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdcf5f4e-4113-42a0-9ac1-fc37743b5ee5" containerName="revoke-keys" Jul 14 12:07:29.516909 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.516854 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="571260b0-aa8a-41c8-860f-387924e08fc9" containerName="revoke-keys" Jul 14 12:07:29.523341 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.523319 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.526970 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.526950 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-91627dc1-openshift-default-dockercfg-9jf97\"" Jul 14 12:07:29.539453 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.539417 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp"] Jul 14 12:07:29.631150 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.631113 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-data\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.631351 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.631177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-token\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.631351 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.631285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6shv\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-kube-api-access-s6shv\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.631476 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.631399 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-certs\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.631476 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.631455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-envoy\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.631585 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.631490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-socket\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.631585 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.631525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-credential-socket\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.631585 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.631551 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-podinfo\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.631585 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.631577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac662ac-e2f5-458b-b3a2-50fc358b9444-istiod-ca-cert\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732126 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732103 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-envoy\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732126 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-socket\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732301 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-credential-socket\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732301 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-podinfo\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732301 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732195 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac662ac-e2f5-458b-b3a2-50fc358b9444-istiod-ca-cert\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732301 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-data\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732301 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-token\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732301 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6shv\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-kube-api-access-s6shv\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732609 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-certs\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732609 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-credential-socket\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732725 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-certs\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732859 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732832 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-socket\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732967 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732872 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-data\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.732967 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.732939 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac662ac-e2f5-458b-b3a2-50fc358b9444-istiod-ca-cert\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.734951 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.734928 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-envoy\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.735061 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.735037 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-podinfo\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.741677 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.741656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6shv\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-kube-api-access-s6shv\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.741751 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.741722 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-token\") pod \"e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.841043 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.840988 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:29.975140 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.975115 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp"] Jul 14 12:07:29.975747 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:07:29.975723 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac662ac_e2f5_458b_b3a2_50fc358b9444.slice/crio-6f0193a6e3ab83a3d2e05fb88c52aa9fd23757128d15c55046e1adab66cd6986 WatchSource:0}: Error finding container 6f0193a6e3ab83a3d2e05fb88c52aa9fd23757128d15c55046e1adab66cd6986: Status 404 returned error can't find the container with id 6f0193a6e3ab83a3d2e05fb88c52aa9fd23757128d15c55046e1adab66cd6986 Jul 14 12:07:29.977840 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.977808 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:07:29.977941 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.977871 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:07:29.977941 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:29.977899 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:07:30.473425 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:30.473385 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" event={"ID":"aac662ac-e2f5-458b-b3a2-50fc358b9444","Type":"ContainerStarted","Data":"f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58"} Jul 14 12:07:30.473794 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:30.473450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" event={"ID":"aac662ac-e2f5-458b-b3a2-50fc358b9444","Type":"ContainerStarted","Data":"6f0193a6e3ab83a3d2e05fb88c52aa9fd23757128d15c55046e1adab66cd6986"} Jul 14 12:07:30.498634 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:30.498588 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" podStartSLOduration=1.498574363 podStartE2EDuration="1.498574363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:07:30.49672405 +0000 UTC m=+1783.572578642" watchObservedRunningTime="2026-07-14 12:07:30.498574363 +0000 UTC m=+1783.574428951" Jul 14 12:07:30.842037 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:30.841958 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:31.846260 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:31.846232 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:32.484968 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:32.484940 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:32.485765 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:32.485743 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:34.461241 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.461218 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:07:34.497749 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.497567 2562 generic.go:356] "Generic (PLEG): container finished" podID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerID="d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758" exitCode=0 Jul 14 12:07:34.497749 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.497602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" event={"ID":"19973f19-f8e9-42aa-ab5b-6da94fc5f375","Type":"ContainerDied","Data":"d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758"} Jul 14 12:07:34.497749 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.497631 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" Jul 14 12:07:34.497749 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.497657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t" event={"ID":"19973f19-f8e9-42aa-ab5b-6da94fc5f375","Type":"ContainerDied","Data":"115e6672668b39e0ec954e2d0cd6da97c36d951021ba6ece765087d754f89c8a"} Jul 14 12:07:34.497749 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.497691 2562 scope.go:117] "RemoveContainer" containerID="d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758" Jul 14 12:07:34.507559 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.507530 2562 scope.go:117] "RemoveContainer" containerID="355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e" Jul 14 12:07:34.518304 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.518270 2562 scope.go:117] "RemoveContainer" containerID="d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758" Jul 14 12:07:34.518802 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:07:34.518776 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758\": container with ID starting with d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758 not found: ID does not exist" containerID="d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758" Jul 14 12:07:34.518900 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.518804 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758"} err="failed to get container status \"d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758\": rpc error: code = NotFound desc = could not find container \"d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758\": container with ID starting with d84aa92c388be36b060aad502e55a53f7a9e16ffc305e84d895db0f689a58758 not found: ID does not exist" Jul 14 12:07:34.518900 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.518820 2562 scope.go:117] "RemoveContainer" containerID="355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e" Jul 14 12:07:34.519107 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:07:34.519072 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e\": container with ID starting with 355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e not found: ID does not exist" containerID="355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e" Jul 14 12:07:34.519161 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.519112 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e"} err="failed to get container status \"355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e\": rpc error: code = NotFound desc = could not find container \"355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e\": container with ID starting with 355f91d993eac181316642028147492b799f345ff917353502bc3ca42967ce6e not found: ID does not exist" Jul 14 12:07:34.573322 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573302 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-data\") pod \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " Jul 14 12:07:34.573439 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573375 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-envoy\") pod \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " Jul 14 12:07:34.573503 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573436 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istiod-ca-cert\") pod \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " Jul 14 12:07:34.573503 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573464 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-podinfo\") pod \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " Jul 14 12:07:34.573503 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573497 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-credential-socket\") pod \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " Jul 14 12:07:34.573664 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573522 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-token\") pod \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " Jul 14 12:07:34.573664 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573612 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-socket\") pod \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " Jul 14 12:07:34.573755 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573679 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-certs\") pod \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " Jul 14 12:07:34.573811 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573755 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgg2p\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-kube-api-access-fgg2p\") pod \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\" (UID: \"19973f19-f8e9-42aa-ab5b-6da94fc5f375\") " Jul 14 12:07:34.573811 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573786 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19973f19-f8e9-42aa-ab5b-6da94fc5f375" (UID: "19973f19-f8e9-42aa-ab5b-6da94fc5f375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:07:34.573931 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573874 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19973f19-f8e9-42aa-ab5b-6da94fc5f375" (UID: "19973f19-f8e9-42aa-ab5b-6da94fc5f375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:07:34.573988 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.573962 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19973f19-f8e9-42aa-ab5b-6da94fc5f375" (UID: "19973f19-f8e9-42aa-ab5b-6da94fc5f375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:07:34.574245 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.574140 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19973f19-f8e9-42aa-ab5b-6da94fc5f375" (UID: "19973f19-f8e9-42aa-ab5b-6da94fc5f375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:07:34.574245 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.574147 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:34.574245 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.574183 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:34.574245 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.574198 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:34.574245 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.574213 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-data" (OuterVolumeSpecName: "istio-data") pod "19973f19-f8e9-42aa-ab5b-6da94fc5f375" (UID: "19973f19-f8e9-42aa-ab5b-6da94fc5f375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:07:34.575547 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.575519 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19973f19-f8e9-42aa-ab5b-6da94fc5f375" (UID: "19973f19-f8e9-42aa-ab5b-6da94fc5f375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:07:34.575646 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.575601 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19973f19-f8e9-42aa-ab5b-6da94fc5f375" (UID: "19973f19-f8e9-42aa-ab5b-6da94fc5f375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:07:34.575965 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.575947 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-token" (OuterVolumeSpecName: "istio-token") pod "19973f19-f8e9-42aa-ab5b-6da94fc5f375" (UID: "19973f19-f8e9-42aa-ab5b-6da94fc5f375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:07:34.576045 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.576027 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-kube-api-access-fgg2p" (OuterVolumeSpecName: "kube-api-access-fgg2p") pod "19973f19-f8e9-42aa-ab5b-6da94fc5f375" (UID: "19973f19-f8e9-42aa-ab5b-6da94fc5f375"). InnerVolumeSpecName "kube-api-access-fgg2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:07:34.675559 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.675536 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:34.675559 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.675557 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgg2p\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-kube-api-access-fgg2p\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:34.675751 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.675567 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:34.675751 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.675576 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:34.675751 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.675585 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:34.675751 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.675593 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19973f19-f8e9-42aa-ab5b-6da94fc5f375-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:34.824444 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.824419 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t"] Jul 14 12:07:34.829440 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:34.829418 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0135d283-openshift-default-7b57b5fd4c-s6g4t"] Jul 14 12:07:35.053377 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.053353 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh"] Jul 14 12:07:35.053682 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.053670 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" Jul 14 12:07:35.053727 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.053684 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" Jul 14 12:07:35.053727 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.053696 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" Jul 14 12:07:35.053727 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.053701 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" Jul 14 12:07:35.053827 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.053765 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" Jul 14 12:07:35.056585 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.056545 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" Jul 14 12:07:35.059647 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.059629 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:07:35.059742 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.059660 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:07:35.066963 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.066943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh"] Jul 14 12:07:35.179599 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.179573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c35ace-a462-4146-9e4d-6627c1e40666-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh\" (UID: \"a0c35ace-a462-4146-9e4d-6627c1e40666\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" Jul 14 12:07:35.280941 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.280914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c35ace-a462-4146-9e4d-6627c1e40666-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh\" (UID: \"a0c35ace-a462-4146-9e4d-6627c1e40666\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" Jul 14 12:07:35.281527 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.281510 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c35ace-a462-4146-9e4d-6627c1e40666-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh\" (UID: \"a0c35ace-a462-4146-9e4d-6627c1e40666\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" Jul 14 12:07:35.367227 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.367176 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" Jul 14 12:07:35.480202 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.480171 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" path="/var/lib/kubelet/pods/19973f19-f8e9-42aa-ab5b-6da94fc5f375/volumes" Jul 14 12:07:35.497468 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:35.497441 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh"] Jul 14 12:07:35.498450 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:07:35.498422 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c35ace_a462_4146_9e4d_6627c1e40666.slice/crio-21d3f78fd9d86d90f3ab952a524f7a30955b1540ea4622539af606022ce235e2 WatchSource:0}: Error finding container 21d3f78fd9d86d90f3ab952a524f7a30955b1540ea4622539af606022ce235e2: Status 404 returned error can't find the container with id 21d3f78fd9d86d90f3ab952a524f7a30955b1540ea4622539af606022ce235e2 Jul 14 12:07:36.511582 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:36.511549 2562 generic.go:356] "Generic (PLEG): container finished" podID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerID="4711006ef36d4166e50df433da576793aa4781c9097614d7f2f16895eb058af7" exitCode=7 Jul 14 12:07:36.511947 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:36.511620 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" event={"ID":"a0c35ace-a462-4146-9e4d-6627c1e40666","Type":"ContainerDied","Data":"4711006ef36d4166e50df433da576793aa4781c9097614d7f2f16895eb058af7"} Jul 14 12:07:36.511947 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:36.511666 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" event={"ID":"a0c35ace-a462-4146-9e4d-6627c1e40666","Type":"ContainerStarted","Data":"21d3f78fd9d86d90f3ab952a524f7a30955b1540ea4622539af606022ce235e2"} Jul 14 12:07:36.511947 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:36.511740 2562 scope.go:117] "RemoveContainer" containerID="4711006ef36d4166e50df433da576793aa4781c9097614d7f2f16895eb058af7" Jul 14 12:07:37.518326 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:37.518288 2562 generic.go:356] "Generic (PLEG): container finished" podID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerID="c51dc781d5f6b7e0673b195656bc5f09d4553cae4e7f67d0cb4c4eb5c9eab35e" exitCode=7 Jul 14 12:07:37.518727 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:37.518357 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" event={"ID":"a0c35ace-a462-4146-9e4d-6627c1e40666","Type":"ContainerDied","Data":"c51dc781d5f6b7e0673b195656bc5f09d4553cae4e7f67d0cb4c4eb5c9eab35e"} Jul 14 12:07:37.518727 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:37.518415 2562 scope.go:117] "RemoveContainer" containerID="4711006ef36d4166e50df433da576793aa4781c9097614d7f2f16895eb058af7" Jul 14 12:07:37.518727 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:37.518515 2562 scope.go:117] "RemoveContainer" containerID="c51dc781d5f6b7e0673b195656bc5f09d4553cae4e7f67d0cb4c4eb5c9eab35e" Jul 14 12:07:37.518727 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:07:37.518702 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh_odh-ai-gateway-infra(a0c35ace-a462-4146-9e4d-6627c1e40666)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" Jul 14 12:07:38.524872 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:38.524847 2562 scope.go:117] "RemoveContainer" containerID="c51dc781d5f6b7e0673b195656bc5f09d4553cae4e7f67d0cb4c4eb5c9eab35e" Jul 14 12:07:38.525257 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:07:38.525011 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh_odh-ai-gateway-infra(a0c35ace-a462-4146-9e4d-6627c1e40666)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" Jul 14 12:07:47.457015 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.456984 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:07:47.460826 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.460807 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:07:47.460944 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.460873 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:07:47.464745 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.464728 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:07:47.487277 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.487240 2562 patch_prober.go:28] interesting pod/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 14 12:07:47.487378 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.487307 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 14 12:07:47.585983 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.585952 2562 generic.go:356] "Generic (PLEG): container finished" podID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerID="f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58" exitCode=0 Jul 14 12:07:47.586116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.586004 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" event={"ID":"aac662ac-e2f5-458b-b3a2-50fc358b9444","Type":"ContainerDied","Data":"f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58"} Jul 14 12:07:47.586376 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.586362 2562 scope.go:117] "RemoveContainer" containerID="f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58" Jul 14 12:07:47.587282 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.587252 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:07:47.587338 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.587321 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:07:47.587373 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:47.587350 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:07:48.593604 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:48.593563 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" event={"ID":"aac662ac-e2f5-458b-b3a2-50fc358b9444","Type":"ContainerStarted","Data":"f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5"} Jul 14 12:07:48.842220 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:48.842177 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:48.842220 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:48.842215 2562 patch_prober.go:28] interesting pod/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 14 12:07:48.842496 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:48.842261 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 14 12:07:49.841578 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:49.841541 2562 patch_prober.go:28] interesting pod/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 14 12:07:49.842193 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:49.842160 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 14 12:07:50.842477 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:50.842424 2562 patch_prober.go:28] interesting pod/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 14 12:07:50.842987 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:50.842518 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 14 12:07:51.474480 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:51.474446 2562 scope.go:117] "RemoveContainer" containerID="c51dc781d5f6b7e0673b195656bc5f09d4553cae4e7f67d0cb4c4eb5c9eab35e" Jul 14 12:07:51.848102 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:51.845148 2562 patch_prober.go:28] interesting pod/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 14 12:07:51.848102 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:51.845215 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 14 12:07:52.617579 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:52.617541 2562 generic.go:356] "Generic (PLEG): container finished" podID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerID="1a4c64f9ce1bf413e867ddca432116f5c6c614efddcff6b80a8491071a47c1b2" exitCode=0 Jul 14 12:07:52.617758 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:52.617611 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" event={"ID":"a0c35ace-a462-4146-9e4d-6627c1e40666","Type":"ContainerDied","Data":"1a4c64f9ce1bf413e867ddca432116f5c6c614efddcff6b80a8491071a47c1b2"} Jul 14 12:07:52.617758 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:52.617699 2562 scope.go:117] "RemoveContainer" containerID="c51dc781d5f6b7e0673b195656bc5f09d4553cae4e7f67d0cb4c4eb5c9eab35e" Jul 14 12:07:52.845989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:52.845963 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:52.846319 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:52.846301 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:52.847053 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:52.847034 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:07:53.662940 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:53.662908 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh"] Jul 14 12:07:53.765987 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:53.765964 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" Jul 14 12:07:53.840583 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:53.840557 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c35ace-a462-4146-9e4d-6627c1e40666-maas-api-service-ca\") pod \"a0c35ace-a462-4146-9e4d-6627c1e40666\" (UID: \"a0c35ace-a462-4146-9e4d-6627c1e40666\") " Jul 14 12:07:53.840935 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:53.840916 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c35ace-a462-4146-9e4d-6627c1e40666-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0c35ace-a462-4146-9e4d-6627c1e40666" (UID: "a0c35ace-a462-4146-9e4d-6627c1e40666"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:07:53.941750 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:53.941663 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c35ace-a462-4146-9e4d-6627c1e40666-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:07:54.631455 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:54.631428 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" Jul 14 12:07:54.631668 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:54.631479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh" event={"ID":"a0c35ace-a462-4146-9e4d-6627c1e40666","Type":"ContainerDied","Data":"21d3f78fd9d86d90f3ab952a524f7a30955b1540ea4622539af606022ce235e2"} Jul 14 12:07:54.631668 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:54.631535 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21d3f78fd9d86d90f3ab952a524f7a30955b1540ea4622539af606022ce235e2" Jul 14 12:07:54.667430 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:54.667402 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh"] Jul 14 12:07:54.674146 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:54.674123 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-jn8lh"] Jul 14 12:07:55.479621 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:07:55.479540 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" path="/var/lib/kubelet/pods/a0c35ace-a462-4146-9e4d-6627c1e40666/volumes" Jul 14 12:08:25.104622 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.104586 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h"] Jul 14 12:08:25.105116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.104902 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerName="revoke-keys" Jul 14 12:08:25.105116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.104929 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerName="revoke-keys" Jul 14 12:08:25.105116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.104943 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerName="revoke-keys" Jul 14 12:08:25.105116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.104949 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerName="revoke-keys" Jul 14 12:08:25.105116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.104956 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerName="revoke-keys" Jul 14 12:08:25.105116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.104962 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerName="revoke-keys" Jul 14 12:08:25.105116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.105024 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerName="revoke-keys" Jul 14 12:08:25.105116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.105032 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerName="revoke-keys" Jul 14 12:08:25.105116 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.105039 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="19973f19-f8e9-42aa-ab5b-6da94fc5f375" containerName="istio-proxy" Jul 14 12:08:25.108010 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.107993 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" Jul 14 12:08:25.110903 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.110883 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:08:25.111004 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.110910 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:08:25.118321 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.118299 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h"] Jul 14 12:08:25.191227 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.191199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab97a9c-a1ef-495e-bf1c-42de59162ed6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h\" (UID: \"6ab97a9c-a1ef-495e-bf1c-42de59162ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" Jul 14 12:08:25.292324 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.292300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab97a9c-a1ef-495e-bf1c-42de59162ed6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h\" (UID: \"6ab97a9c-a1ef-495e-bf1c-42de59162ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" Jul 14 12:08:25.292890 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.292874 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab97a9c-a1ef-495e-bf1c-42de59162ed6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h\" (UID: \"6ab97a9c-a1ef-495e-bf1c-42de59162ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" Jul 14 12:08:25.418951 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.418895 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" Jul 14 12:08:25.556243 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.556217 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h"] Jul 14 12:08:25.557280 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:08:25.557252 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab97a9c_a1ef_495e_bf1c_42de59162ed6.slice/crio-e354b7390d7327781c15dcfb6105950b9fe7d93ba064ae7425aada642e6b6519 WatchSource:0}: Error finding container e354b7390d7327781c15dcfb6105950b9fe7d93ba064ae7425aada642e6b6519: Status 404 returned error can't find the container with id e354b7390d7327781c15dcfb6105950b9fe7d93ba064ae7425aada642e6b6519 Jul 14 12:08:25.800866 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:25.800840 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" event={"ID":"6ab97a9c-a1ef-495e-bf1c-42de59162ed6","Type":"ContainerStarted","Data":"e354b7390d7327781c15dcfb6105950b9fe7d93ba064ae7425aada642e6b6519"} Jul 14 12:08:26.808224 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:26.808188 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ab97a9c-a1ef-495e-bf1c-42de59162ed6" containerID="bf8fdc94acd8ebfdd6297268ecc06ed0f906c9d237fa070e2a87c51df7872b08" exitCode=0 Jul 14 12:08:26.808668 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:26.808275 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" event={"ID":"6ab97a9c-a1ef-495e-bf1c-42de59162ed6","Type":"ContainerDied","Data":"bf8fdc94acd8ebfdd6297268ecc06ed0f906c9d237fa070e2a87c51df7872b08"} Jul 14 12:08:27.940958 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:27.940933 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" Jul 14 12:08:28.014336 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:28.014310 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab97a9c-a1ef-495e-bf1c-42de59162ed6-maas-api-service-ca\") pod \"6ab97a9c-a1ef-495e-bf1c-42de59162ed6\" (UID: \"6ab97a9c-a1ef-495e-bf1c-42de59162ed6\") " Jul 14 12:08:28.014638 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:28.014616 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab97a9c-a1ef-495e-bf1c-42de59162ed6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ab97a9c-a1ef-495e-bf1c-42de59162ed6" (UID: "6ab97a9c-a1ef-495e-bf1c-42de59162ed6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:08:28.115349 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:28.115286 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab97a9c-a1ef-495e-bf1c-42de59162ed6-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:28.820691 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:28.820659 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" Jul 14 12:08:28.820691 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:28.820677 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h" event={"ID":"6ab97a9c-a1ef-495e-bf1c-42de59162ed6","Type":"ContainerDied","Data":"e354b7390d7327781c15dcfb6105950b9fe7d93ba064ae7425aada642e6b6519"} Jul 14 12:08:28.820932 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:28.820725 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e354b7390d7327781c15dcfb6105950b9fe7d93ba064ae7425aada642e6b6519" Jul 14 12:08:50.412808 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:50.412769 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp"] Jul 14 12:08:50.413252 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:50.412985 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" containerID="cri-o://f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5" gracePeriod=30 Jul 14 12:08:52.848388 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:52.848342 2562 patch_prober.go:28] interesting pod/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 12:08:52.848780 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:52.848399 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 12:08:55.668493 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.668466 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:08:55.736587 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736556 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-credential-socket\") pod \"aac662ac-e2f5-458b-b3a2-50fc358b9444\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " Jul 14 12:08:55.736729 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736651 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac662ac-e2f5-458b-b3a2-50fc358b9444-istiod-ca-cert\") pod \"aac662ac-e2f5-458b-b3a2-50fc358b9444\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " Jul 14 12:08:55.736729 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736681 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-certs\") pod \"aac662ac-e2f5-458b-b3a2-50fc358b9444\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " Jul 14 12:08:55.736729 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736712 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-token\") pod \"aac662ac-e2f5-458b-b3a2-50fc358b9444\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " Jul 14 12:08:55.736904 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736738 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-envoy\") pod \"aac662ac-e2f5-458b-b3a2-50fc358b9444\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " Jul 14 12:08:55.736904 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736782 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-socket\") pod \"aac662ac-e2f5-458b-b3a2-50fc358b9444\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " Jul 14 12:08:55.736904 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736838 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6shv\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-kube-api-access-s6shv\") pod \"aac662ac-e2f5-458b-b3a2-50fc358b9444\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " Jul 14 12:08:55.736904 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736843 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aac662ac-e2f5-458b-b3a2-50fc358b9444" (UID: "aac662ac-e2f5-458b-b3a2-50fc358b9444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:08:55.736904 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736875 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-data\") pod \"aac662ac-e2f5-458b-b3a2-50fc358b9444\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " Jul 14 12:08:55.737205 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.736925 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-podinfo\") pod \"aac662ac-e2f5-458b-b3a2-50fc358b9444\" (UID: \"aac662ac-e2f5-458b-b3a2-50fc358b9444\") " Jul 14 12:08:55.737205 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.737096 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aac662ac-e2f5-458b-b3a2-50fc358b9444" (UID: "aac662ac-e2f5-458b-b3a2-50fc358b9444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:08:55.737205 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.737131 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac662ac-e2f5-458b-b3a2-50fc358b9444-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aac662ac-e2f5-458b-b3a2-50fc358b9444" (UID: "aac662ac-e2f5-458b-b3a2-50fc358b9444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:08:55.737368 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.737341 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aac662ac-e2f5-458b-b3a2-50fc358b9444" (UID: "aac662ac-e2f5-458b-b3a2-50fc358b9444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:08:55.737425 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.737390 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:55.737425 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.737417 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac662ac-e2f5-458b-b3a2-50fc358b9444-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:55.737530 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.737433 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:55.737914 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.737893 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-data" (OuterVolumeSpecName: "istio-data") pod "aac662ac-e2f5-458b-b3a2-50fc358b9444" (UID: "aac662ac-e2f5-458b-b3a2-50fc358b9444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:08:55.739057 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.739028 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-kube-api-access-s6shv" (OuterVolumeSpecName: "kube-api-access-s6shv") pod "aac662ac-e2f5-458b-b3a2-50fc358b9444" (UID: "aac662ac-e2f5-458b-b3a2-50fc358b9444"). InnerVolumeSpecName "kube-api-access-s6shv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:08:55.739333 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.739309 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aac662ac-e2f5-458b-b3a2-50fc358b9444" (UID: "aac662ac-e2f5-458b-b3a2-50fc358b9444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:08:55.739389 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.739317 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-token" (OuterVolumeSpecName: "istio-token") pod "aac662ac-e2f5-458b-b3a2-50fc358b9444" (UID: "aac662ac-e2f5-458b-b3a2-50fc358b9444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:08:55.739529 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.739509 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aac662ac-e2f5-458b-b3a2-50fc358b9444" (UID: "aac662ac-e2f5-458b-b3a2-50fc358b9444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:08:55.838580 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.838547 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6shv\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-kube-api-access-s6shv\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:55.838580 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.838578 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:55.838580 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.838587 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:55.838791 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.838595 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:55.838791 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.838607 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:55.838791 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.838615 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac662ac-e2f5-458b-b3a2-50fc358b9444-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:08:55.978253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.978175 2562 generic.go:356] "Generic (PLEG): container finished" podID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerID="f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5" exitCode=0 Jul 14 12:08:55.978253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.978242 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" Jul 14 12:08:55.978417 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.978265 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" event={"ID":"aac662ac-e2f5-458b-b3a2-50fc358b9444","Type":"ContainerDied","Data":"f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5"} Jul 14 12:08:55.978417 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.978321 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp" event={"ID":"aac662ac-e2f5-458b-b3a2-50fc358b9444","Type":"ContainerDied","Data":"6f0193a6e3ab83a3d2e05fb88c52aa9fd23757128d15c55046e1adab66cd6986"} Jul 14 12:08:55.978417 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.978344 2562 scope.go:117] "RemoveContainer" containerID="f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5" Jul 14 12:08:55.987815 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.987800 2562 scope.go:117] "RemoveContainer" containerID="f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58" Jul 14 12:08:55.995990 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.995975 2562 scope.go:117] "RemoveContainer" containerID="f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5" Jul 14 12:08:55.996262 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:08:55.996242 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5\": container with ID starting with f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5 not found: ID does not exist" containerID="f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5" Jul 14 12:08:55.996350 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.996269 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5"} err="failed to get container status \"f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5\": rpc error: code = NotFound desc = could not find container \"f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5\": container with ID starting with f72a32fb1b897f14d77d4fb777e82ac8a77b0a9a0ab6e9db45e2fe88b68fcab5 not found: ID does not exist" Jul 14 12:08:55.996350 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.996286 2562 scope.go:117] "RemoveContainer" containerID="f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58" Jul 14 12:08:55.996504 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:08:55.996485 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58\": container with ID starting with f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58 not found: ID does not exist" containerID="f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58" Jul 14 12:08:55.996544 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:55.996519 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58"} err="failed to get container status \"f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58\": rpc error: code = NotFound desc = could not find container \"f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58\": container with ID starting with f89d6f5c2aa1f1f3cfd66a8fc62c3d841491e60c9591d12b4ecf82baf464ac58 not found: ID does not exist" Jul 14 12:08:56.003159 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:56.003140 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp"] Jul 14 12:08:56.007955 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:56.007935 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-91627dc1-openshift-default-7698c7b9f9-g9bhp"] Jul 14 12:08:57.480741 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:08:57.480709 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" path="/var/lib/kubelet/pods/aac662ac-e2f5-458b-b3a2-50fc358b9444/volumes" Jul 14 12:09:47.527743 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:09:47.527711 2562 scope.go:117] "RemoveContainer" containerID="bfea0c9a7f85f741c33762b4dbbd25dd28b02f2ad02198df1496712c4a564df5" Jul 14 12:10:32.959309 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.959269 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt"] Jul 14 12:10:32.959830 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.959590 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab97a9c-a1ef-495e-bf1c-42de59162ed6" containerName="revoke-keys" Jul 14 12:10:32.959830 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.959602 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab97a9c-a1ef-495e-bf1c-42de59162ed6" containerName="revoke-keys" Jul 14 12:10:32.959830 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.959611 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" Jul 14 12:10:32.959830 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.959617 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" Jul 14 12:10:32.959830 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.959679 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab97a9c-a1ef-495e-bf1c-42de59162ed6" containerName="revoke-keys" Jul 14 12:10:32.959830 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.959687 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" Jul 14 12:10:32.959830 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.959693 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c35ace-a462-4146-9e4d-6627c1e40666" containerName="revoke-keys" Jul 14 12:10:32.959830 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.959698 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" Jul 14 12:10:32.962567 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.962549 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:32.965510 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:32.965491 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5374a9b0-openshift-default-dockercfg-4thhc\"" Jul 14 12:10:33.050886 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.050857 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt"] Jul 14 12:10:33.088009 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.087985 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-certs\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.088159 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.088039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-socket\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.088159 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.088120 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-podinfo\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.088284 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.088173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istiod-ca-cert\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.088284 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.088203 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-data\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.088284 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.088227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzj2w\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-kube-api-access-wzj2w\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.088284 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.088254 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-token\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.088449 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.088294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-credential-socket\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.088449 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.088323 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-envoy\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.188974 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.188947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-certs\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189162 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-socket\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189162 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-podinfo\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189162 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istiod-ca-cert\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189162 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-data\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189162 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzj2w\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-kube-api-access-wzj2w\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189162 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-token\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189542 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-credential-socket\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189542 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-envoy\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189542 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189409 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-certs\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189542 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-socket\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189741 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-data\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189790 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189751 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-credential-socket\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.189829 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.189807 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istiod-ca-cert\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.191523 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.191505 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-envoy\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.192049 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.192030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-podinfo\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.199345 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.199321 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-token\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.199833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.199812 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzj2w\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-kube-api-access-wzj2w\") pod \"e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.272117 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.272095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:33.408694 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.408659 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt"] Jul 14 12:10:33.409784 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:10:33.409758 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47f8b13_66c1_45f6_b8c7_6b6fff0a89cf.slice/crio-075ac189eb3fcb7a506318111c60395a924a4e4b8e4b7872a5228e338eb17532 WatchSource:0}: Error finding container 075ac189eb3fcb7a506318111c60395a924a4e4b8e4b7872a5228e338eb17532: Status 404 returned error can't find the container with id 075ac189eb3fcb7a506318111c60395a924a4e4b8e4b7872a5228e338eb17532 Jul 14 12:10:33.412095 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.412047 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:10:33.412165 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.412150 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:10:33.412206 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.412194 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:10:33.499264 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.499233 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" event={"ID":"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf","Type":"ContainerStarted","Data":"0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705"} Jul 14 12:10:33.499387 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.499287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" event={"ID":"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf","Type":"ContainerStarted","Data":"075ac189eb3fcb7a506318111c60395a924a4e4b8e4b7872a5228e338eb17532"} Jul 14 12:10:33.527105 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:33.526995 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" podStartSLOduration=1.526973877 podStartE2EDuration="1.526973877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:10:33.522587889 +0000 UTC m=+1966.598442490" watchObservedRunningTime="2026-07-14 12:10:33.526973877 +0000 UTC m=+1966.602828471" Jul 14 12:10:34.272770 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:34.272738 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:34.277320 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:34.277301 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:34.504377 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:34.504340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:34.505338 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:34.505316 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:47.603310 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:47.603277 2562 generic.go:356] "Generic (PLEG): container finished" podID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerID="0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705" exitCode=0 Jul 14 12:10:47.603702 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:47.603368 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" event={"ID":"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf","Type":"ContainerDied","Data":"0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705"} Jul 14 12:10:47.603785 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:47.603768 2562 scope.go:117] "RemoveContainer" containerID="0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705" Jul 14 12:10:47.604819 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:47.604790 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:10:47.604910 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:47.604857 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:10:47.604910 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:47.604888 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:10:48.611454 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:48.611416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" event={"ID":"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf","Type":"ContainerStarted","Data":"a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e"} Jul 14 12:10:48.905224 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:48.905137 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd"] Jul 14 12:10:48.905482 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:48.905470 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" Jul 14 12:10:48.905527 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:48.905483 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac662ac-e2f5-458b-b3a2-50fc358b9444" containerName="istio-proxy" Jul 14 12:10:48.907579 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:48.907563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" Jul 14 12:10:48.910465 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:48.910434 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:10:48.910465 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:48.910453 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:10:48.919650 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:48.919625 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd"] Jul 14 12:10:49.030512 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.030471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a2867e-2956-4159-8fdc-1e2ab336a5c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd\" (UID: \"f4a2867e-2956-4159-8fdc-1e2ab336a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" Jul 14 12:10:49.131460 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.131422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a2867e-2956-4159-8fdc-1e2ab336a5c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd\" (UID: \"f4a2867e-2956-4159-8fdc-1e2ab336a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" Jul 14 12:10:49.132066 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.132046 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a2867e-2956-4159-8fdc-1e2ab336a5c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd\" (UID: \"f4a2867e-2956-4159-8fdc-1e2ab336a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" Jul 14 12:10:49.217211 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.217131 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" Jul 14 12:10:49.272435 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.272401 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:49.273425 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.273401 2562 patch_prober.go:28] interesting pod/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 14 12:10:49.273512 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.273464 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 14 12:10:49.403577 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.403552 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd"] Jul 14 12:10:49.405005 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:10:49.404976 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a2867e_2956_4159_8fdc_1e2ab336a5c8.slice/crio-7ab900026e47f737eb658ce12d76ab035e8aa8088f4a3e19995137e492da5941 WatchSource:0}: Error finding container 7ab900026e47f737eb658ce12d76ab035e8aa8088f4a3e19995137e492da5941: Status 404 returned error can't find the container with id 7ab900026e47f737eb658ce12d76ab035e8aa8088f4a3e19995137e492da5941 Jul 14 12:10:49.505071 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.505043 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:49.618808 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:49.618779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" event={"ID":"f4a2867e-2956-4159-8fdc-1e2ab336a5c8","Type":"ContainerStarted","Data":"7ab900026e47f737eb658ce12d76ab035e8aa8088f4a3e19995137e492da5941"} Jul 14 12:10:50.272581 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:50.272549 2562 patch_prober.go:28] interesting pod/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 14 12:10:50.272780 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:50.272610 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 14 12:10:50.626033 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:50.625946 2562 generic.go:356] "Generic (PLEG): container finished" podID="f4a2867e-2956-4159-8fdc-1e2ab336a5c8" containerID="2da745f6d42ca61713e2cd4e99f5ae4cacd2d9e4ad4e5259d03e5df4cb2fcd50" exitCode=0 Jul 14 12:10:50.626453 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:50.626023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" event={"ID":"f4a2867e-2956-4159-8fdc-1e2ab336a5c8","Type":"ContainerDied","Data":"2da745f6d42ca61713e2cd4e99f5ae4cacd2d9e4ad4e5259d03e5df4cb2fcd50"} Jul 14 12:10:51.273351 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:51.273317 2562 patch_prober.go:28] interesting pod/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 14 12:10:51.273521 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:51.273370 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 14 12:10:51.791675 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:51.791650 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" Jul 14 12:10:51.876821 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:51.876788 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a2867e-2956-4159-8fdc-1e2ab336a5c8-maas-api-service-ca\") pod \"f4a2867e-2956-4159-8fdc-1e2ab336a5c8\" (UID: \"f4a2867e-2956-4159-8fdc-1e2ab336a5c8\") " Jul 14 12:10:51.877178 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:51.877154 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a2867e-2956-4159-8fdc-1e2ab336a5c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4a2867e-2956-4159-8fdc-1e2ab336a5c8" (UID: "f4a2867e-2956-4159-8fdc-1e2ab336a5c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:10:51.977962 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:51.977936 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a2867e-2956-4159-8fdc-1e2ab336a5c8-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:10:52.273245 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:52.273170 2562 patch_prober.go:28] interesting pod/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 14 12:10:52.273245 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:52.273221 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 14 12:10:52.641705 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:52.641604 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" event={"ID":"f4a2867e-2956-4159-8fdc-1e2ab336a5c8","Type":"ContainerDied","Data":"7ab900026e47f737eb658ce12d76ab035e8aa8088f4a3e19995137e492da5941"} Jul 14 12:10:52.641705 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:52.641703 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab900026e47f737eb658ce12d76ab035e8aa8088f4a3e19995137e492da5941" Jul 14 12:10:52.641941 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:52.641770 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd" Jul 14 12:10:53.276448 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:53.276418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:10:53.277491 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:10:53.277470 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:11:14.710366 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:14.710334 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt"] Jul 14 12:11:14.710900 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:14.710582 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" containerID="cri-o://a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e" gracePeriod=30 Jul 14 12:11:19.963462 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:19.963435 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:11:20.011996 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.011961 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-data\") pod \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " Jul 14 12:11:20.012175 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012006 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istiod-ca-cert\") pod \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " Jul 14 12:11:20.012175 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012024 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzj2w\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-kube-api-access-wzj2w\") pod \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " Jul 14 12:11:20.012175 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012043 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-token\") pod \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " Jul 14 12:11:20.012175 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012106 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-credential-socket\") pod \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " Jul 14 12:11:20.012175 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012131 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-podinfo\") pod \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " Jul 14 12:11:20.012463 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012195 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-envoy\") pod \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " Jul 14 12:11:20.012463 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012259 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-certs\") pod \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " Jul 14 12:11:20.012463 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012428 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" (UID: "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:11:20.012608 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012539 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:11:20.012667 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012592 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" (UID: "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:11:20.012723 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012663 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" (UID: "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:11:20.012919 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.012899 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" (UID: "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:11:20.014334 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.014314 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" (UID: "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:11:20.014895 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.014856 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" (UID: "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:11:20.014895 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.014887 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" (UID: "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:11:20.015004 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.014955 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-kube-api-access-wzj2w" (OuterVolumeSpecName: "kube-api-access-wzj2w") pod "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" (UID: "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf"). InnerVolumeSpecName "kube-api-access-wzj2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:11:20.113337 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.113299 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-socket\") pod \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\" (UID: \"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf\") " Jul 14 12:11:20.113518 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.113505 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:11:20.113565 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.113521 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:11:20.113565 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.113529 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:11:20.113565 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.113537 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzj2w\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-kube-api-access-wzj2w\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:11:20.113565 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.113544 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:11:20.113565 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.113552 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:11:20.113565 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.113560 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:11:20.113744 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.113562 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" (UID: "b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:11:20.214582 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.214543 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:11:20.793896 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.793862 2562 generic.go:356] "Generic (PLEG): container finished" podID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerID="a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e" exitCode=0 Jul 14 12:11:20.794066 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.793915 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" Jul 14 12:11:20.794066 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.793945 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" event={"ID":"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf","Type":"ContainerDied","Data":"a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e"} Jul 14 12:11:20.794066 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.794002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt" event={"ID":"b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf","Type":"ContainerDied","Data":"075ac189eb3fcb7a506318111c60395a924a4e4b8e4b7872a5228e338eb17532"} Jul 14 12:11:20.794066 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.794022 2562 scope.go:117] "RemoveContainer" containerID="a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e" Jul 14 12:11:20.803517 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.803503 2562 scope.go:117] "RemoveContainer" containerID="0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705" Jul 14 12:11:20.811053 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.811037 2562 scope.go:117] "RemoveContainer" containerID="a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e" Jul 14 12:11:20.811368 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:11:20.811350 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e\": container with ID starting with a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e not found: ID does not exist" containerID="a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e" Jul 14 12:11:20.811433 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.811375 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e"} err="failed to get container status \"a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e\": rpc error: code = NotFound desc = could not find container \"a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e\": container with ID starting with a0cd8601989a52a55b720f1dc6465bdc2a36b6bab74f0781bc9c4d03de8e764e not found: ID does not exist" Jul 14 12:11:20.811433 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.811391 2562 scope.go:117] "RemoveContainer" containerID="0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705" Jul 14 12:11:20.811632 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:11:20.811614 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705\": container with ID starting with 0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705 not found: ID does not exist" containerID="0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705" Jul 14 12:11:20.811670 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.811638 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705"} err="failed to get container status \"0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705\": rpc error: code = NotFound desc = could not find container \"0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705\": container with ID starting with 0bde0fa97f878290603cc55f821f02136854e83fcfdc1eae44f57db9d1001705 not found: ID does not exist" Jul 14 12:11:20.818925 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.818906 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt"] Jul 14 12:11:20.824978 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:20.824958 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5374a9b0-openshift-default-7fb9cd5555-k64zt"] Jul 14 12:11:21.479619 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:21.479586 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" path="/var/lib/kubelet/pods/b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf/volumes" Jul 14 12:11:36.550489 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.550456 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn"] Jul 14 12:11:36.550937 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.550922 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" Jul 14 12:11:36.550989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.550941 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" Jul 14 12:11:36.550989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.550957 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a2867e-2956-4159-8fdc-1e2ab336a5c8" containerName="revoke-keys" Jul 14 12:11:36.550989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.550965 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a2867e-2956-4159-8fdc-1e2ab336a5c8" containerName="revoke-keys" Jul 14 12:11:36.551110 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.551068 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" Jul 14 12:11:36.551110 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.551094 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a2867e-2956-4159-8fdc-1e2ab336a5c8" containerName="revoke-keys" Jul 14 12:11:36.551110 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.551106 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" Jul 14 12:11:36.553731 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.553712 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.556779 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.556752 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c52b78c6-openshift-default-dockercfg-ncshl\"" Jul 14 12:11:36.574906 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.574883 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn"] Jul 14 12:11:36.639229 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.639199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563236-fedd-4280-96da-d04dc6220262-istiod-ca-cert\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.639369 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.639236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-certs\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.639369 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.639270 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-istio-token\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.639369 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.639314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4h4\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-kube-api-access-lx4h4\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.639369 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.639334 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-credential-socket\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.639591 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.639401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-data\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.639591 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.639435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-envoy\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.639591 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.639452 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563236-fedd-4280-96da-d04dc6220262-istio-podinfo\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.639591 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.639472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-socket\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.740647 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.740619 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-data\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.740647 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.740650 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-envoy\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.740871 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.740667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563236-fedd-4280-96da-d04dc6220262-istio-podinfo\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.740871 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.740687 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-socket\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.740871 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.740737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563236-fedd-4280-96da-d04dc6220262-istiod-ca-cert\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.740871 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.740762 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-certs\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.740871 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.740785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-istio-token\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.740871 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.740828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4h4\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-kube-api-access-lx4h4\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.740871 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.740853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-credential-socket\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.741259 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.741115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-data\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.741259 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.741234 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-socket\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.741364 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.741335 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-certs\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.741480 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.741464 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-credential-socket\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.741763 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.741740 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563236-fedd-4280-96da-d04dc6220262-istiod-ca-cert\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.742883 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.742865 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-envoy\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.743214 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.743196 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563236-fedd-4280-96da-d04dc6220262-istio-podinfo\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.752246 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.752227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-istio-token\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.752800 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.752777 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4h4\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-kube-api-access-lx4h4\") pod \"e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:36.863935 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:36.863876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:37.010230 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:37.010204 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn"] Jul 14 12:11:37.010873 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:11:37.010840 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e563236_fedd_4280_96da_d04dc6220262.slice/crio-fc7c06143316d7130c8859a8ddc8dbc6e51652e3bdde66de20384b28ec1c9b8b WatchSource:0}: Error finding container fc7c06143316d7130c8859a8ddc8dbc6e51652e3bdde66de20384b28ec1c9b8b: Status 404 returned error can't find the container with id fc7c06143316d7130c8859a8ddc8dbc6e51652e3bdde66de20384b28ec1c9b8b Jul 14 12:11:37.012899 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:37.012864 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:11:37.012993 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:37.012927 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:11:37.012993 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:37.012955 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:11:37.889995 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:37.889955 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" event={"ID":"6e563236-fedd-4280-96da-d04dc6220262","Type":"ContainerStarted","Data":"81e70b221a120adc02b1bbfa3d1f2095ef509104dfa8ad6cedd26b9e53fb1745"} Jul 14 12:11:37.890411 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:37.890017 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" event={"ID":"6e563236-fedd-4280-96da-d04dc6220262","Type":"ContainerStarted","Data":"fc7c06143316d7130c8859a8ddc8dbc6e51652e3bdde66de20384b28ec1c9b8b"} Jul 14 12:11:37.919746 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:37.919699 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" podStartSLOduration=1.91968686 podStartE2EDuration="1.91968686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:11:37.918542417 +0000 UTC m=+2030.994397003" watchObservedRunningTime="2026-07-14 12:11:37.91968686 +0000 UTC m=+2030.995541448" Jul 14 12:11:38.864026 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:38.863999 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:38.868510 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:38.868484 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:38.895940 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:38.895914 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:38.896787 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:38.896764 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:47.560432 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:47.560391 2562 scope.go:117] "RemoveContainer" containerID="8a15d978d3ba9b734eb09ade6db82defb6f90975c7daebdf0d44661395b9ea54" Jul 14 12:11:53.897340 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:53.897278 2562 patch_prober.go:28] interesting pod/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 14 12:11:53.897340 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:53.897329 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 14 12:11:53.994691 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:53.994658 2562 generic.go:356] "Generic (PLEG): container finished" podID="6e563236-fedd-4280-96da-d04dc6220262" containerID="81e70b221a120adc02b1bbfa3d1f2095ef509104dfa8ad6cedd26b9e53fb1745" exitCode=0 Jul 14 12:11:53.994849 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:53.994733 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" event={"ID":"6e563236-fedd-4280-96da-d04dc6220262","Type":"ContainerDied","Data":"81e70b221a120adc02b1bbfa3d1f2095ef509104dfa8ad6cedd26b9e53fb1745"} Jul 14 12:11:53.995138 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:53.995122 2562 scope.go:117] "RemoveContainer" containerID="81e70b221a120adc02b1bbfa3d1f2095ef509104dfa8ad6cedd26b9e53fb1745" Jul 14 12:11:53.996193 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:53.996165 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:11:53.996298 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:53.996225 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:11:53.996298 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:53.996257 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:11:55.002069 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:55.002036 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" event={"ID":"6e563236-fedd-4280-96da-d04dc6220262","Type":"ContainerStarted","Data":"81474fc060f18e1db8fc912f9b5d4189527cfacd562d6f72f7b1384312e6f07f"} Jul 14 12:11:55.864403 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:55.864362 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:55.864693 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:55.864669 2562 patch_prober.go:28] interesting pod/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 14 12:11:55.864744 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:55.864721 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 14 12:11:56.864588 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:56.864547 2562 patch_prober.go:28] interesting pod/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 14 12:11:56.865009 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:56.864611 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 14 12:11:57.864477 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:57.864440 2562 patch_prober.go:28] interesting pod/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 14 12:11:57.864674 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:57.864503 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 14 12:11:58.864832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:58.864794 2562 patch_prober.go:28] interesting pod/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 14 12:11:58.865268 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:58.864851 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 14 12:11:59.868240 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:59.868200 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:59.868714 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:59.868568 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:11:59.869176 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:11:59.869155 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:12:17.727009 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.726926 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj"] Jul 14 12:12:17.727493 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.727459 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" Jul 14 12:12:17.727493 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.727477 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47f8b13-66c1-45f6-b8c7-6b6fff0a89cf" containerName="istio-proxy" Jul 14 12:12:17.729827 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.729807 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" Jul 14 12:12:17.732813 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.732796 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:12:17.732813 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.732807 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:12:17.742117 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.742070 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj"] Jul 14 12:12:17.822576 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.822537 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ab9b59-4bab-459e-9f5f-1cc5f146c60e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj\" (UID: \"35ab9b59-4bab-459e-9f5f-1cc5f146c60e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" Jul 14 12:12:17.923615 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.923581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ab9b59-4bab-459e-9f5f-1cc5f146c60e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj\" (UID: \"35ab9b59-4bab-459e-9f5f-1cc5f146c60e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" Jul 14 12:12:17.924246 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:17.924227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ab9b59-4bab-459e-9f5f-1cc5f146c60e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj\" (UID: \"35ab9b59-4bab-459e-9f5f-1cc5f146c60e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" Jul 14 12:12:18.040822 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:18.040734 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" Jul 14 12:12:18.179028 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:18.179003 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj"] Jul 14 12:12:18.179644 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:12:18.179619 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ab9b59_4bab_459e_9f5f_1cc5f146c60e.slice/crio-19603c6fd69e0a72779c0ae3040592c7bd8afd76e73fcbcf311f0208a1f80175 WatchSource:0}: Error finding container 19603c6fd69e0a72779c0ae3040592c7bd8afd76e73fcbcf311f0208a1f80175: Status 404 returned error can't find the container with id 19603c6fd69e0a72779c0ae3040592c7bd8afd76e73fcbcf311f0208a1f80175 Jul 14 12:12:19.128409 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:19.128316 2562 generic.go:356] "Generic (PLEG): container finished" podID="35ab9b59-4bab-459e-9f5f-1cc5f146c60e" containerID="3edf4f265c25a1a480cc74e80fc70b0ea134c749a491f13f9604656404d8cca8" exitCode=0 Jul 14 12:12:19.128768 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:19.128408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" event={"ID":"35ab9b59-4bab-459e-9f5f-1cc5f146c60e","Type":"ContainerDied","Data":"3edf4f265c25a1a480cc74e80fc70b0ea134c749a491f13f9604656404d8cca8"} Jul 14 12:12:19.128768 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:19.128468 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" event={"ID":"35ab9b59-4bab-459e-9f5f-1cc5f146c60e","Type":"ContainerStarted","Data":"19603c6fd69e0a72779c0ae3040592c7bd8afd76e73fcbcf311f0208a1f80175"} Jul 14 12:12:20.262819 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:20.262796 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" Jul 14 12:12:20.345376 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:20.345342 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ab9b59-4bab-459e-9f5f-1cc5f146c60e-maas-api-service-ca\") pod \"35ab9b59-4bab-459e-9f5f-1cc5f146c60e\" (UID: \"35ab9b59-4bab-459e-9f5f-1cc5f146c60e\") " Jul 14 12:12:20.345703 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:20.345679 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ab9b59-4bab-459e-9f5f-1cc5f146c60e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35ab9b59-4bab-459e-9f5f-1cc5f146c60e" (UID: "35ab9b59-4bab-459e-9f5f-1cc5f146c60e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:12:20.445946 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:20.445858 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35ab9b59-4bab-459e-9f5f-1cc5f146c60e-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:12:21.140378 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:21.140344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" event={"ID":"35ab9b59-4bab-459e-9f5f-1cc5f146c60e","Type":"ContainerDied","Data":"19603c6fd69e0a72779c0ae3040592c7bd8afd76e73fcbcf311f0208a1f80175"} Jul 14 12:12:21.140378 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:21.140370 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj" Jul 14 12:12:21.140579 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:21.140395 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19603c6fd69e0a72779c0ae3040592c7bd8afd76e73fcbcf311f0208a1f80175" Jul 14 12:12:47.496484 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:47.496455 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:12:47.504734 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:47.504706 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:12:47.510460 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:47.510440 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:12:47.513955 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:47.513935 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:12:47.586615 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:12:47.586594 2562 scope.go:117] "RemoveContainer" containerID="a7871537ce0927824b31fe38d65f246d0c31c80d2f9e0dd4cde739dca4e8b782" Jul 14 12:13:35.369988 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:35.369952 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn"] Jul 14 12:13:35.370545 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:35.370200 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" containerID="cri-o://81474fc060f18e1db8fc912f9b5d4189527cfacd562d6f72f7b1384312e6f07f" gracePeriod=30 Jul 14 12:13:40.579212 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.579176 2562 generic.go:356] "Generic (PLEG): container finished" podID="6e563236-fedd-4280-96da-d04dc6220262" containerID="81474fc060f18e1db8fc912f9b5d4189527cfacd562d6f72f7b1384312e6f07f" exitCode=0 Jul 14 12:13:40.579575 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.579251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" event={"ID":"6e563236-fedd-4280-96da-d04dc6220262","Type":"ContainerDied","Data":"81474fc060f18e1db8fc912f9b5d4189527cfacd562d6f72f7b1384312e6f07f"} Jul 14 12:13:40.579575 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.579320 2562 scope.go:117] "RemoveContainer" containerID="81e70b221a120adc02b1bbfa3d1f2095ef509104dfa8ad6cedd26b9e53fb1745" Jul 14 12:13:40.621822 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.621802 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:13:40.723259 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723237 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-istio-token\") pod \"6e563236-fedd-4280-96da-d04dc6220262\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " Jul 14 12:13:40.723447 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723267 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-certs\") pod \"6e563236-fedd-4280-96da-d04dc6220262\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " Jul 14 12:13:40.723447 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723287 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-credential-socket\") pod \"6e563236-fedd-4280-96da-d04dc6220262\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " Jul 14 12:13:40.723447 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723356 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563236-fedd-4280-96da-d04dc6220262-istio-podinfo\") pod \"6e563236-fedd-4280-96da-d04dc6220262\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " Jul 14 12:13:40.723447 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723389 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-envoy\") pod \"6e563236-fedd-4280-96da-d04dc6220262\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " Jul 14 12:13:40.723447 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723419 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-data\") pod \"6e563236-fedd-4280-96da-d04dc6220262\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " Jul 14 12:13:40.723447 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723446 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx4h4\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-kube-api-access-lx4h4\") pod \"6e563236-fedd-4280-96da-d04dc6220262\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " Jul 14 12:13:40.723744 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723491 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-socket\") pod \"6e563236-fedd-4280-96da-d04dc6220262\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " Jul 14 12:13:40.723744 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723531 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563236-fedd-4280-96da-d04dc6220262-istiod-ca-cert\") pod \"6e563236-fedd-4280-96da-d04dc6220262\" (UID: \"6e563236-fedd-4280-96da-d04dc6220262\") " Jul 14 12:13:40.723744 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723540 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6e563236-fedd-4280-96da-d04dc6220262" (UID: "6e563236-fedd-4280-96da-d04dc6220262"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:13:40.723744 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723659 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6e563236-fedd-4280-96da-d04dc6220262" (UID: "6e563236-fedd-4280-96da-d04dc6220262"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:13:40.723958 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723828 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:13:40.723958 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723848 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:13:40.723958 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.723917 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e563236-fedd-4280-96da-d04dc6220262-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6e563236-fedd-4280-96da-d04dc6220262" (UID: "6e563236-fedd-4280-96da-d04dc6220262"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:13:40.724349 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.724317 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6e563236-fedd-4280-96da-d04dc6220262" (UID: "6e563236-fedd-4280-96da-d04dc6220262"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:13:40.724517 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.724495 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-data" (OuterVolumeSpecName: "istio-data") pod "6e563236-fedd-4280-96da-d04dc6220262" (UID: "6e563236-fedd-4280-96da-d04dc6220262"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:13:40.725470 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.725448 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e563236-fedd-4280-96da-d04dc6220262" (UID: "6e563236-fedd-4280-96da-d04dc6220262"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:13:40.725697 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.725683 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-kube-api-access-lx4h4" (OuterVolumeSpecName: "kube-api-access-lx4h4") pod "6e563236-fedd-4280-96da-d04dc6220262" (UID: "6e563236-fedd-4280-96da-d04dc6220262"). InnerVolumeSpecName "kube-api-access-lx4h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:13:40.725888 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.725872 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e563236-fedd-4280-96da-d04dc6220262-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6e563236-fedd-4280-96da-d04dc6220262" (UID: "6e563236-fedd-4280-96da-d04dc6220262"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:13:40.725949 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.725880 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6e563236-fedd-4280-96da-d04dc6220262" (UID: "6e563236-fedd-4280-96da-d04dc6220262"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:13:40.825136 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.825111 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:13:40.825136 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.825133 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563236-fedd-4280-96da-d04dc6220262-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:13:40.825267 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.825144 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:13:40.825267 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.825152 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:13:40.825267 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.825160 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx4h4\" (UniqueName: \"kubernetes.io/projected/6e563236-fedd-4280-96da-d04dc6220262-kube-api-access-lx4h4\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:13:40.825267 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.825170 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563236-fedd-4280-96da-d04dc6220262-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:13:40.825267 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:40.825179 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563236-fedd-4280-96da-d04dc6220262-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:13:41.588659 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:41.588630 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" Jul 14 12:13:41.589086 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:41.588659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn" event={"ID":"6e563236-fedd-4280-96da-d04dc6220262","Type":"ContainerDied","Data":"fc7c06143316d7130c8859a8ddc8dbc6e51652e3bdde66de20384b28ec1c9b8b"} Jul 14 12:13:41.589086 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:41.588712 2562 scope.go:117] "RemoveContainer" containerID="81474fc060f18e1db8fc912f9b5d4189527cfacd562d6f72f7b1384312e6f07f" Jul 14 12:13:41.615048 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:41.615025 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn"] Jul 14 12:13:41.629089 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:41.629053 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c52b78c6-openshift-default-5846bbf4c-jjkzn"] Jul 14 12:13:43.479515 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:13:43.479479 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e563236-fedd-4280-96da-d04dc6220262" path="/var/lib/kubelet/pods/6e563236-fedd-4280-96da-d04dc6220262/volumes" Jul 14 12:14:33.768746 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.768712 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb"] Jul 14 12:14:33.769233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.769036 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" Jul 14 12:14:33.769233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.769046 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" Jul 14 12:14:33.769233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.769054 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ab9b59-4bab-459e-9f5f-1cc5f146c60e" containerName="revoke-keys" Jul 14 12:14:33.769233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.769060 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ab9b59-4bab-459e-9f5f-1cc5f146c60e" containerName="revoke-keys" Jul 14 12:14:33.769233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.769089 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" Jul 14 12:14:33.769233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.769095 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" Jul 14 12:14:33.769233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.769155 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" Jul 14 12:14:33.769233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.769162 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e563236-fedd-4280-96da-d04dc6220262" containerName="istio-proxy" Jul 14 12:14:33.769233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.769168 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ab9b59-4bab-459e-9f5f-1cc5f146c60e" containerName="revoke-keys" Jul 14 12:14:33.772056 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.772039 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.775247 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.775229 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ae12a0-openshift-default-dockercfg-f5k42\"" Jul 14 12:14:33.786938 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.786912 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb"] Jul 14 12:14:33.884574 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.884543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-credential-socket\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.884574 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.884576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-certs\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.884833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.884602 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istiod-ca-cert\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.884833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.884660 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-data\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.884833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.884724 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-envoy\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.884833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.884770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-socket\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.884833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.884821 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-token\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.885055 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.884869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsx58\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-kube-api-access-zsx58\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.885055 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.884929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-podinfo\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985384 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-podinfo\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985535 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-credential-socket\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985535 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985412 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-certs\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985535 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istiod-ca-cert\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985535 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-data\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985535 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-envoy\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-socket\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985608 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-token\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985656 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsx58\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-kube-api-access-zsx58\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.985814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985795 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-credential-socket\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.986032 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-data\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.986032 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.985896 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-certs\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.986248 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.986223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-socket\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.986532 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.986513 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istiod-ca-cert\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.987878 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.987859 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-envoy\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.988032 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.988012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-podinfo\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.996369 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.996348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-token\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:33.996589 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:33.996572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsx58\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-kube-api-access-zsx58\") pod \"e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:34.083000 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:34.082950 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:34.235195 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:34.235104 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb"] Jul 14 12:14:34.238516 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:14:34.238488 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e72a415_e4b1_4078_bcab_efa8d4d645a5.slice/crio-0cf619758bde8d60a2042b5236faab5a4b46a3d8cc24c6ec17a007eb6f9cb525 WatchSource:0}: Error finding container 0cf619758bde8d60a2042b5236faab5a4b46a3d8cc24c6ec17a007eb6f9cb525: Status 404 returned error can't find the container with id 0cf619758bde8d60a2042b5236faab5a4b46a3d8cc24c6ec17a007eb6f9cb525 Jul 14 12:14:34.241042 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:34.241000 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:14:34.241143 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:34.241116 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:14:34.241220 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:34.241163 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:14:34.879278 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:34.879242 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" event={"ID":"0e72a415-e4b1-4078-bcab-efa8d4d645a5","Type":"ContainerStarted","Data":"61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419"} Jul 14 12:14:34.879742 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:34.879290 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" event={"ID":"0e72a415-e4b1-4078-bcab-efa8d4d645a5","Type":"ContainerStarted","Data":"0cf619758bde8d60a2042b5236faab5a4b46a3d8cc24c6ec17a007eb6f9cb525"} Jul 14 12:14:34.907358 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:34.907296 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" podStartSLOduration=1.907281941 podStartE2EDuration="1.907281941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:14:34.905829351 +0000 UTC m=+2207.981683941" watchObservedRunningTime="2026-07-14 12:14:34.907281941 +0000 UTC m=+2207.983136530" Jul 14 12:14:35.083694 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:35.083663 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:35.088431 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:35.088408 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:35.884601 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:35.884573 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:35.885463 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:35.885441 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:42.158382 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.158347 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962"] Jul 14 12:14:42.162098 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.162060 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.165288 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.165268 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a015ba-openshift-default-dockercfg-z5txg\"" Jul 14 12:14:42.184478 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.184456 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962"] Jul 14 12:14:42.351379 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.351332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-envoy\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.351566 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.351421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-certs\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.351566 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.351482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-data\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.351566 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.351510 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxwpg\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-kube-api-access-pxwpg\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.351566 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.351539 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-token\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.351804 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.351571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-credential-socket\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.351804 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.351596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-podinfo\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.351804 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.351618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istiod-ca-cert\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.351804 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.351676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-socket\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.452817 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.452735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-envoy\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.452817 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.452799 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-certs\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453001 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.452860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-data\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453001 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.452887 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwpg\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-kube-api-access-pxwpg\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453001 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.452915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-token\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453001 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.452946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-credential-socket\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453001 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.452971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-podinfo\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453001 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.452997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istiod-ca-cert\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453354 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.453048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-socket\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453354 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.453299 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-data\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453461 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.453356 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-certs\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453519 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.453455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-credential-socket\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453585 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.453526 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-socket\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.453843 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.453817 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istiod-ca-cert\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.455184 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.455164 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-envoy\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.455601 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.455578 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-podinfo\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.464666 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.464639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-token\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.464973 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.464956 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwpg\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-kube-api-access-pxwpg\") pod \"e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.472804 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.472786 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:42.613100 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.613045 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962"] Jul 14 12:14:42.613526 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:14:42.613506 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0abe9e_6b15_472a_828d_af5eb2ca66b4.slice/crio-3ff6eafc3245f423f02e9c92cbd859b68f1e84fd5c999ceff58da915cf54ad5d WatchSource:0}: Error finding container 3ff6eafc3245f423f02e9c92cbd859b68f1e84fd5c999ceff58da915cf54ad5d: Status 404 returned error can't find the container with id 3ff6eafc3245f423f02e9c92cbd859b68f1e84fd5c999ceff58da915cf54ad5d Jul 14 12:14:42.615635 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.615604 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:14:42.615709 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.615668 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:14:42.615709 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.615697 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:14:42.922925 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.922887 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" event={"ID":"8a0abe9e-6b15-472a-828d-af5eb2ca66b4","Type":"ContainerStarted","Data":"e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4"} Jul 14 12:14:42.923101 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.922951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" event={"ID":"8a0abe9e-6b15-472a-828d-af5eb2ca66b4","Type":"ContainerStarted","Data":"3ff6eafc3245f423f02e9c92cbd859b68f1e84fd5c999ceff58da915cf54ad5d"} Jul 14 12:14:42.957601 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:42.957546 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" podStartSLOduration=0.957526906 podStartE2EDuration="957.526906ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:14:42.953878337 +0000 UTC m=+2216.029732926" watchObservedRunningTime="2026-07-14 12:14:42.957526906 +0000 UTC m=+2216.033381496" Jul 14 12:14:43.473785 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:43.473745 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:43.479785 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:43.479762 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:43.928779 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:43.928748 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:43.929519 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:43.929503 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:14:47.617320 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:47.617288 2562 scope.go:117] "RemoveContainer" containerID="1a4c64f9ce1bf413e867ddca432116f5c6c614efddcff6b80a8491071a47c1b2" Jul 14 12:14:51.926360 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:51.926312 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 12:14:51.926776 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:51.926400 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 12:14:52.990616 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:52.990584 2562 generic.go:356] "Generic (PLEG): container finished" podID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerID="61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419" exitCode=0 Jul 14 12:14:52.991021 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:52.990652 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" event={"ID":"0e72a415-e4b1-4078-bcab-efa8d4d645a5","Type":"ContainerDied","Data":"61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419"} Jul 14 12:14:52.991069 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:52.991053 2562 scope.go:117] "RemoveContainer" containerID="61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419" Jul 14 12:14:52.992049 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:52.992018 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:14:52.992137 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:52.992118 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:14:52.992179 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:52.992161 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:14:53.999325 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:53.999292 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" event={"ID":"0e72a415-e4b1-4078-bcab-efa8d4d645a5","Type":"ContainerStarted","Data":"5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640"} Jul 14 12:14:54.083641 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:54.083556 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:54.083945 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:54.083920 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 14 12:14:54.084171 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:54.084145 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 14 12:14:55.083979 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:55.083538 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 14 12:14:55.083979 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:55.083604 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 14 12:14:56.083947 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:56.083907 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 14 12:14:56.084144 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:56.083981 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 14 12:14:57.084096 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:57.084037 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 14 12:14:57.084270 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:57.084129 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 14 12:14:58.087127 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:58.087088 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:58.087643 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:58.087467 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:58.088277 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:58.088258 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:14:59.273165 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:59.272659 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:40274->10.133.0.58:15021: read: connection reset by peer" start-of-body= Jul 14 12:14:59.273165 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:14:59.272723 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:40274->10.133.0.58:15021: read: connection reset by peer" Jul 14 12:15:00.036700 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.036668 2562 generic.go:356] "Generic (PLEG): container finished" podID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerID="e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4" exitCode=0 Jul 14 12:15:00.036884 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.036746 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" event={"ID":"8a0abe9e-6b15-472a-828d-af5eb2ca66b4","Type":"ContainerDied","Data":"e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4"} Jul 14 12:15:00.037258 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.037244 2562 scope.go:117] "RemoveContainer" containerID="e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4" Jul 14 12:15:00.038169 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.038144 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:15:00.038225 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.038208 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:15:00.038264 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.038240 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:15:00.141356 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.141322 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4"] Jul 14 12:15:00.143831 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.143811 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4" Jul 14 12:15:00.146746 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.146724 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:15:00.155244 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.155213 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4" Jul 14 12:15:00.157389 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.157332 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4"] Jul 14 12:15:00.305179 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:00.305147 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4"] Jul 14 12:15:00.306018 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:15:00.305981 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde3b0a8_e7d6_4543_a96b_74fc525ca119.slice/crio-9ed9fcbfd32a99c2da53858b2b1e09dfd81b35c17c461401a1e2c708dfcc909a WatchSource:0}: Error finding container 9ed9fcbfd32a99c2da53858b2b1e09dfd81b35c17c461401a1e2c708dfcc909a: Status 404 returned error can't find the container with id 9ed9fcbfd32a99c2da53858b2b1e09dfd81b35c17c461401a1e2c708dfcc909a Jul 14 12:15:01.049598 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:01.049550 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" event={"ID":"8a0abe9e-6b15-472a-828d-af5eb2ca66b4","Type":"ContainerStarted","Data":"bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66"} Jul 14 12:15:01.052412 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:01.052373 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4" event={"ID":"dde3b0a8-e7d6-4543-a96b-74fc525ca119","Type":"ContainerStarted","Data":"e2196c88055365fa41866ffb9fc219e32ca411dd46d813229ea69e131ab8c9ff"} Jul 14 12:15:01.052573 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:01.052438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4" event={"ID":"dde3b0a8-e7d6-4543-a96b-74fc525ca119","Type":"ContainerStarted","Data":"9ed9fcbfd32a99c2da53858b2b1e09dfd81b35c17c461401a1e2c708dfcc909a"} Jul 14 12:15:01.116239 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:01.116177 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4" podStartSLOduration=1.116154477 podStartE2EDuration="1.116154477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:15:01.113754867 +0000 UTC m=+2234.189609457" watchObservedRunningTime="2026-07-14 12:15:01.116154477 +0000 UTC m=+2234.192009067" Jul 14 12:15:01.473089 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:01.473058 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:15:01.473512 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:01.473296 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 14 12:15:01.473512 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:01.473354 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 14 12:15:02.062981 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:02.062947 2562 generic.go:356] "Generic (PLEG): container finished" podID="dde3b0a8-e7d6-4543-a96b-74fc525ca119" containerID="e2196c88055365fa41866ffb9fc219e32ca411dd46d813229ea69e131ab8c9ff" exitCode=0 Jul 14 12:15:02.063210 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:02.063046 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4" event={"ID":"dde3b0a8-e7d6-4543-a96b-74fc525ca119","Type":"ContainerDied","Data":"e2196c88055365fa41866ffb9fc219e32ca411dd46d813229ea69e131ab8c9ff"} Jul 14 12:15:02.474072 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:02.474043 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 14 12:15:02.474485 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:02.474151 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 14 12:15:03.214647 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:03.214619 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4" Jul 14 12:15:03.473926 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:03.473892 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 14 12:15:03.474143 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:03.473948 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 14 12:15:04.075882 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:04.075838 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4" event={"ID":"dde3b0a8-e7d6-4543-a96b-74fc525ca119","Type":"ContainerDied","Data":"9ed9fcbfd32a99c2da53858b2b1e09dfd81b35c17c461401a1e2c708dfcc909a"} Jul 14 12:15:04.075882 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:04.075859 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4" Jul 14 12:15:04.075882 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:04.075891 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ed9fcbfd32a99c2da53858b2b1e09dfd81b35c17c461401a1e2c708dfcc909a" Jul 14 12:15:04.473658 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:04.473626 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 14 12:15:04.473818 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:04.473691 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 14 12:15:05.479224 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:05.479193 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:15:05.479592 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:05.479460 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:15:05.480072 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:05.480052 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:15:54.526673 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:54.526634 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4"] Jul 14 12:15:54.530950 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:54.530927 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ae12a0-29733855-sl4v4"] Jul 14 12:15:55.478851 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:15:55.478807 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dde3b0a8-e7d6-4543-a96b-74fc525ca119" path="/var/lib/kubelet/pods/dde3b0a8-e7d6-4543-a96b-74fc525ca119/volumes" Jul 14 12:16:09.886315 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:09.886269 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb"] Jul 14 12:16:09.886774 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:09.886526 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" containerID="cri-o://5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640" gracePeriod=30 Jul 14 12:16:10.261658 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.261627 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp"] Jul 14 12:16:10.261969 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.261957 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dde3b0a8-e7d6-4543-a96b-74fc525ca119" containerName="cleanup" Jul 14 12:16:10.262011 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.261971 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dde3b0a8-e7d6-4543-a96b-74fc525ca119" containerName="cleanup" Jul 14 12:16:10.262045 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.262041 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dde3b0a8-e7d6-4543-a96b-74fc525ca119" containerName="cleanup" Jul 14 12:16:10.265357 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.265340 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" Jul 14 12:16:10.268456 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.268437 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:16:10.268553 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.268481 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:16:10.275767 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.275747 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp"] Jul 14 12:16:10.320471 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.320444 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b16b7-9a6e-400d-9e46-009eb07e670d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp\" (UID: \"752b16b7-9a6e-400d-9e46-009eb07e670d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" Jul 14 12:16:10.421260 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.421239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b16b7-9a6e-400d-9e46-009eb07e670d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp\" (UID: \"752b16b7-9a6e-400d-9e46-009eb07e670d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" Jul 14 12:16:10.421823 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.421806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b16b7-9a6e-400d-9e46-009eb07e670d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp\" (UID: \"752b16b7-9a6e-400d-9e46-009eb07e670d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" Jul 14 12:16:10.576282 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.576222 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" Jul 14 12:16:10.711779 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:10.711749 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp"] Jul 14 12:16:10.712743 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:16:10.712715 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752b16b7_9a6e_400d_9e46_009eb07e670d.slice/crio-3a417f22c7e54e0ca0e0120c7b7651815d79c4b7967530ee69ed5d6e2c7573a9 WatchSource:0}: Error finding container 3a417f22c7e54e0ca0e0120c7b7651815d79c4b7967530ee69ed5d6e2c7573a9: Status 404 returned error can't find the container with id 3a417f22c7e54e0ca0e0120c7b7651815d79c4b7967530ee69ed5d6e2c7573a9 Jul 14 12:16:11.446980 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:11.446943 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" event={"ID":"752b16b7-9a6e-400d-9e46-009eb07e670d","Type":"ContainerStarted","Data":"6874d32d0d3a189005338089b5c108d3f65a9843b88d007f9698fd3b9458f47b"} Jul 14 12:16:11.447318 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:11.447009 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" event={"ID":"752b16b7-9a6e-400d-9e46-009eb07e670d","Type":"ContainerStarted","Data":"3a417f22c7e54e0ca0e0120c7b7651815d79c4b7967530ee69ed5d6e2c7573a9"} Jul 14 12:16:11.469395 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:11.469350 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" podStartSLOduration=1.46933792 podStartE2EDuration="1.46933792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:16:11.468873093 +0000 UTC m=+2304.544727682" watchObservedRunningTime="2026-07-14 12:16:11.46933792 +0000 UTC m=+2304.545192515" Jul 14 12:16:12.459882 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:12.459847 2562 generic.go:356] "Generic (PLEG): container finished" podID="752b16b7-9a6e-400d-9e46-009eb07e670d" containerID="6874d32d0d3a189005338089b5c108d3f65a9843b88d007f9698fd3b9458f47b" exitCode=0 Jul 14 12:16:12.460253 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:12.459907 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" event={"ID":"752b16b7-9a6e-400d-9e46-009eb07e670d","Type":"ContainerDied","Data":"6874d32d0d3a189005338089b5c108d3f65a9843b88d007f9698fd3b9458f47b"} Jul 14 12:16:13.089305 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:13.089275 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 12:16:13.089451 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:13.089330 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 12:16:13.594772 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:13.594747 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" Jul 14 12:16:13.646646 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:13.646618 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b16b7-9a6e-400d-9e46-009eb07e670d-maas-api-service-ca\") pod \"752b16b7-9a6e-400d-9e46-009eb07e670d\" (UID: \"752b16b7-9a6e-400d-9e46-009eb07e670d\") " Jul 14 12:16:13.646951 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:13.646927 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752b16b7-9a6e-400d-9e46-009eb07e670d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "752b16b7-9a6e-400d-9e46-009eb07e670d" (UID: "752b16b7-9a6e-400d-9e46-009eb07e670d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:16:13.747876 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:13.747852 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752b16b7-9a6e-400d-9e46-009eb07e670d-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:14.472171 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:14.472142 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" Jul 14 12:16:14.472336 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:14.472189 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp" event={"ID":"752b16b7-9a6e-400d-9e46-009eb07e670d","Type":"ContainerDied","Data":"3a417f22c7e54e0ca0e0120c7b7651815d79c4b7967530ee69ed5d6e2c7573a9"} Jul 14 12:16:14.472336 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:14.472241 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a417f22c7e54e0ca0e0120c7b7651815d79c4b7967530ee69ed5d6e2c7573a9" Jul 14 12:16:15.141972 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.141951 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:16:15.259126 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259101 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsx58\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-kube-api-access-zsx58\") pod \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " Jul 14 12:16:15.259265 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259150 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-token\") pod \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " Jul 14 12:16:15.259265 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259168 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-podinfo\") pod \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " Jul 14 12:16:15.259265 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259192 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-envoy\") pod \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " Jul 14 12:16:15.259265 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259238 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-certs\") pod \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " Jul 14 12:16:15.259485 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259289 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-credential-socket\") pod \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " Jul 14 12:16:15.259485 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259306 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istiod-ca-cert\") pod \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " Jul 14 12:16:15.259485 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259325 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-socket\") pod \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " Jul 14 12:16:15.259485 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259352 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-data\") pod \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\" (UID: \"0e72a415-e4b1-4078-bcab-efa8d4d645a5\") " Jul 14 12:16:15.259689 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259574 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e72a415-e4b1-4078-bcab-efa8d4d645a5" (UID: "0e72a415-e4b1-4078-bcab-efa8d4d645a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:15.259774 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259746 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e72a415-e4b1-4078-bcab-efa8d4d645a5" (UID: "0e72a415-e4b1-4078-bcab-efa8d4d645a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:15.259971 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259793 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e72a415-e4b1-4078-bcab-efa8d4d645a5" (UID: "0e72a415-e4b1-4078-bcab-efa8d4d645a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:15.259971 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.259918 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e72a415-e4b1-4078-bcab-efa8d4d645a5" (UID: "0e72a415-e4b1-4078-bcab-efa8d4d645a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:16:15.260267 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.260235 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e72a415-e4b1-4078-bcab-efa8d4d645a5" (UID: "0e72a415-e4b1-4078-bcab-efa8d4d645a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:15.261481 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.261457 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e72a415-e4b1-4078-bcab-efa8d4d645a5" (UID: "0e72a415-e4b1-4078-bcab-efa8d4d645a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:16:15.261481 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.261469 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-kube-api-access-zsx58" (OuterVolumeSpecName: "kube-api-access-zsx58") pod "0e72a415-e4b1-4078-bcab-efa8d4d645a5" (UID: "0e72a415-e4b1-4078-bcab-efa8d4d645a5"). InnerVolumeSpecName "kube-api-access-zsx58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:16:15.261606 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.261594 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e72a415-e4b1-4078-bcab-efa8d4d645a5" (UID: "0e72a415-e4b1-4078-bcab-efa8d4d645a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:15.261644 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.261593 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e72a415-e4b1-4078-bcab-efa8d4d645a5" (UID: "0e72a415-e4b1-4078-bcab-efa8d4d645a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:16:15.360051 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.360029 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:15.360051 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.360051 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:15.360182 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.360061 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsx58\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-kube-api-access-zsx58\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:15.360182 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.360071 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:15.360182 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.360104 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:15.360182 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.360118 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:15.360182 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.360130 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:15.360182 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.360142 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e72a415-e4b1-4078-bcab-efa8d4d645a5-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:15.360182 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.360152 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e72a415-e4b1-4078-bcab-efa8d4d645a5-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:15.480363 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.480340 2562 generic.go:356] "Generic (PLEG): container finished" podID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerID="5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640" exitCode=0 Jul 14 12:16:15.480480 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.480385 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" Jul 14 12:16:15.480480 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.480433 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" event={"ID":"0e72a415-e4b1-4078-bcab-efa8d4d645a5","Type":"ContainerDied","Data":"5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640"} Jul 14 12:16:15.480597 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.480491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb" event={"ID":"0e72a415-e4b1-4078-bcab-efa8d4d645a5","Type":"ContainerDied","Data":"0cf619758bde8d60a2042b5236faab5a4b46a3d8cc24c6ec17a007eb6f9cb525"} Jul 14 12:16:15.480597 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.480525 2562 scope.go:117] "RemoveContainer" containerID="5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640" Jul 14 12:16:15.490477 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.490455 2562 scope.go:117] "RemoveContainer" containerID="61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419" Jul 14 12:16:15.498267 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.498252 2562 scope.go:117] "RemoveContainer" containerID="5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640" Jul 14 12:16:15.498499 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:16:15.498480 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640\": container with ID starting with 5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640 not found: ID does not exist" containerID="5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640" Jul 14 12:16:15.498575 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.498512 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640"} err="failed to get container status \"5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640\": rpc error: code = NotFound desc = could not find container \"5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640\": container with ID starting with 5da910ae3a1a6ae89011274cdcea177f5e214f5e58c213526503f054795ef640 not found: ID does not exist" Jul 14 12:16:15.498575 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.498535 2562 scope.go:117] "RemoveContainer" containerID="61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419" Jul 14 12:16:15.498777 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:16:15.498760 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419\": container with ID starting with 61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419 not found: ID does not exist" containerID="61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419" Jul 14 12:16:15.498821 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.498782 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419"} err="failed to get container status \"61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419\": rpc error: code = NotFound desc = could not find container \"61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419\": container with ID starting with 61c3852ac0f139c17ffca6f055d34a0f98ec302421d95bccddb8fbe9b75d6419 not found: ID does not exist" Jul 14 12:16:15.512992 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.512947 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb"] Jul 14 12:16:15.518424 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:15.518403 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ae12a0-openshift-default-77c4776785-gn6pb"] Jul 14 12:16:17.480592 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:17.480546 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" path="/var/lib/kubelet/pods/0e72a415-e4b1-4078-bcab-efa8d4d645a5/volumes" Jul 14 12:16:36.110013 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:36.109980 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962"] Jul 14 12:16:36.110541 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:36.110231 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" containerID="cri-o://bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66" gracePeriod=30 Jul 14 12:16:37.028608 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.028578 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls"] Jul 14 12:16:37.029098 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.029068 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752b16b7-9a6e-400d-9e46-009eb07e670d" containerName="revoke-keys" Jul 14 12:16:37.029155 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.029101 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="752b16b7-9a6e-400d-9e46-009eb07e670d" containerName="revoke-keys" Jul 14 12:16:37.029155 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.029139 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" Jul 14 12:16:37.029155 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.029147 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" Jul 14 12:16:37.029250 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.029224 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="752b16b7-9a6e-400d-9e46-009eb07e670d" containerName="revoke-keys" Jul 14 12:16:37.029250 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.029237 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" Jul 14 12:16:37.029250 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.029246 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" Jul 14 12:16:37.032696 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.032677 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.035682 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.035653 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3ce01d-openshift-default-dockercfg-jc8ld\"" Jul 14 12:16:37.048231 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.048206 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls"] Jul 14 12:16:37.125196 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.125164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-socket\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.125576 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.125205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-credential-socket\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.125576 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.125244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-data\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.125576 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.125309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-certs\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.125576 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.125343 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istiod-ca-cert\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.125576 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.125369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4sb\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-kube-api-access-nh4sb\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.125576 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.125406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-token\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.125576 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.125445 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-envoy\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.125576 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.125520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-podinfo\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226141 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-podinfo\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226314 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-socket\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226314 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-credential-socket\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226314 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-data\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226478 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-certs\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226478 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istiod-ca-cert\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226478 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4sb\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-kube-api-access-nh4sb\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226630 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-token\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226630 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226546 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-envoy\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226739 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226716 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-data\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-socket\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226881 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226818 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-certs\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.226934 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.226918 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-credential-socket\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.227108 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.227071 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istiod-ca-cert\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.228653 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.228630 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-envoy\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.228745 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.228731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-podinfo\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.240684 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.240663 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4sb\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-kube-api-access-nh4sb\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.247640 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.247621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-token\") pod \"e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.343101 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.343029 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:37.483147 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.483125 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls"] Jul 14 12:16:37.484445 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:16:37.484424 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131bd90b_6f48_4fa8_8e06_067eb7cfc683.slice/crio-165b76b9de914e023953102a2dc05d11053ecb5c4419cebd7a1dc322c23b7fd1 WatchSource:0}: Error finding container 165b76b9de914e023953102a2dc05d11053ecb5c4419cebd7a1dc322c23b7fd1: Status 404 returned error can't find the container with id 165b76b9de914e023953102a2dc05d11053ecb5c4419cebd7a1dc322c23b7fd1 Jul 14 12:16:37.486451 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.486415 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:16:37.486539 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.486479 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:16:37.486539 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.486512 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:16:37.604042 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.603959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" event={"ID":"131bd90b-6f48-4fa8-8e06-067eb7cfc683","Type":"ContainerStarted","Data":"b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584"} Jul 14 12:16:37.604216 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.604050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" event={"ID":"131bd90b-6f48-4fa8-8e06-067eb7cfc683","Type":"ContainerStarted","Data":"165b76b9de914e023953102a2dc05d11053ecb5c4419cebd7a1dc322c23b7fd1"} Jul 14 12:16:37.650188 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:37.650104 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" podStartSLOduration=0.650070156 podStartE2EDuration="650.070156ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:16:37.646994495 +0000 UTC m=+2330.722849088" watchObservedRunningTime="2026-07-14 12:16:37.650070156 +0000 UTC m=+2330.725924748" Jul 14 12:16:38.344126 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:38.344097 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:38.348996 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:38.348975 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:38.610237 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:38.610142 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:38.611150 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:38.611133 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:41.353974 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.353948 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:16:41.461387 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461315 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-podinfo\") pod \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " Jul 14 12:16:41.461387 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461349 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-socket\") pod \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " Jul 14 12:16:41.461387 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461370 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-envoy\") pod \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " Jul 14 12:16:41.461387 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461389 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxwpg\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-kube-api-access-pxwpg\") pod \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " Jul 14 12:16:41.461711 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461521 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istiod-ca-cert\") pod \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " Jul 14 12:16:41.461711 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461615 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-token\") pod \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " Jul 14 12:16:41.461711 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461645 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-data\") pod \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " Jul 14 12:16:41.461711 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461682 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-certs\") pod \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " Jul 14 12:16:41.461905 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461697 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a0abe9e-6b15-472a-828d-af5eb2ca66b4" (UID: "8a0abe9e-6b15-472a-828d-af5eb2ca66b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:41.461905 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-credential-socket\") pod \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\" (UID: \"8a0abe9e-6b15-472a-828d-af5eb2ca66b4\") " Jul 14 12:16:41.461905 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.461867 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a0abe9e-6b15-472a-828d-af5eb2ca66b4" (UID: "8a0abe9e-6b15-472a-828d-af5eb2ca66b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:16:41.462156 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.462124 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a0abe9e-6b15-472a-828d-af5eb2ca66b4" (UID: "8a0abe9e-6b15-472a-828d-af5eb2ca66b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:41.462272 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.462255 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:41.462332 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.462305 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:41.462332 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.462320 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:41.462451 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.462331 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a0abe9e-6b15-472a-828d-af5eb2ca66b4" (UID: "8a0abe9e-6b15-472a-828d-af5eb2ca66b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:41.462699 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.462672 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a0abe9e-6b15-472a-828d-af5eb2ca66b4" (UID: "8a0abe9e-6b15-472a-828d-af5eb2ca66b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:41.463728 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.463680 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a0abe9e-6b15-472a-828d-af5eb2ca66b4" (UID: "8a0abe9e-6b15-472a-828d-af5eb2ca66b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:16:41.463728 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.463688 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-kube-api-access-pxwpg" (OuterVolumeSpecName: "kube-api-access-pxwpg") pod "8a0abe9e-6b15-472a-828d-af5eb2ca66b4" (UID: "8a0abe9e-6b15-472a-828d-af5eb2ca66b4"). InnerVolumeSpecName "kube-api-access-pxwpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:16:41.463871 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.463754 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a0abe9e-6b15-472a-828d-af5eb2ca66b4" (UID: "8a0abe9e-6b15-472a-828d-af5eb2ca66b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:16:41.463871 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.463780 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a0abe9e-6b15-472a-828d-af5eb2ca66b4" (UID: "8a0abe9e-6b15-472a-828d-af5eb2ca66b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:16:41.562927 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.562896 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:41.562927 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.562922 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:41.562927 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.562932 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxwpg\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-kube-api-access-pxwpg\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:41.563139 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.562941 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:41.563139 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.562949 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:41.563139 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.562958 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0abe9e-6b15-472a-828d-af5eb2ca66b4-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:16:41.628707 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.628672 2562 generic.go:356] "Generic (PLEG): container finished" podID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerID="bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66" exitCode=0 Jul 14 12:16:41.628866 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.628726 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" Jul 14 12:16:41.628866 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.628726 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" event={"ID":"8a0abe9e-6b15-472a-828d-af5eb2ca66b4","Type":"ContainerDied","Data":"bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66"} Jul 14 12:16:41.628866 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.628787 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962" event={"ID":"8a0abe9e-6b15-472a-828d-af5eb2ca66b4","Type":"ContainerDied","Data":"3ff6eafc3245f423f02e9c92cbd859b68f1e84fd5c999ceff58da915cf54ad5d"} Jul 14 12:16:41.628866 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.628810 2562 scope.go:117] "RemoveContainer" containerID="bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66" Jul 14 12:16:41.640547 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.640531 2562 scope.go:117] "RemoveContainer" containerID="e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4" Jul 14 12:16:41.648793 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.648771 2562 scope.go:117] "RemoveContainer" containerID="bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66" Jul 14 12:16:41.649022 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:16:41.649002 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66\": container with ID starting with bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66 not found: ID does not exist" containerID="bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66" Jul 14 12:16:41.649098 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.649029 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66"} err="failed to get container status \"bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66\": rpc error: code = NotFound desc = could not find container \"bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66\": container with ID starting with bc783b2a5ae29bde1fa9e8af42e8947ff43406e19e9183ea8f721cfe3a664b66 not found: ID does not exist" Jul 14 12:16:41.649098 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.649052 2562 scope.go:117] "RemoveContainer" containerID="e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4" Jul 14 12:16:41.649308 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:16:41.649292 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4\": container with ID starting with e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4 not found: ID does not exist" containerID="e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4" Jul 14 12:16:41.649358 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.649311 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4"} err="failed to get container status \"e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4\": rpc error: code = NotFound desc = could not find container \"e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4\": container with ID starting with e4ac2abea47b85a8b0c528ba35dca75633b93b5410c22935232a724b09feedd4 not found: ID does not exist" Jul 14 12:16:41.653951 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.653932 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962"] Jul 14 12:16:41.666667 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:41.666641 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a015ba-openshift-default-545d68dd95-mk962"] Jul 14 12:16:43.485629 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:43.485596 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" path="/var/lib/kubelet/pods/8a0abe9e-6b15-472a-828d-af5eb2ca66b4/volumes" Jul 14 12:16:51.696167 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:51.696133 2562 generic.go:356] "Generic (PLEG): container finished" podID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerID="b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584" exitCode=0 Jul 14 12:16:51.696553 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:51.696178 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" event={"ID":"131bd90b-6f48-4fa8-8e06-067eb7cfc683","Type":"ContainerDied","Data":"b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584"} Jul 14 12:16:51.696553 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:51.696547 2562 scope.go:117] "RemoveContainer" containerID="b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584" Jul 14 12:16:51.697536 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:51.697507 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:16:51.697600 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:51.697575 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:16:51.697636 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:51.697609 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:16:52.703667 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:52.703637 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" event={"ID":"131bd90b-6f48-4fa8-8e06-067eb7cfc683","Type":"ContainerStarted","Data":"3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d"} Jul 14 12:16:53.344779 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:53.344742 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:53.344982 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:53.344914 2562 patch_prober.go:28] interesting pod/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 14 12:16:53.344982 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:53.344960 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 14 12:16:53.611189 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:53.611070 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:54.344166 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:54.344130 2562 patch_prober.go:28] interesting pod/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 14 12:16:54.344743 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:54.344713 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 14 12:16:55.343911 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:55.343869 2562 patch_prober.go:28] interesting pod/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 14 12:16:55.344232 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:55.344195 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 14 12:16:56.343753 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:56.343715 2562 patch_prober.go:28] interesting pod/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 14 12:16:56.345172 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:56.345141 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 14 12:16:57.347210 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:57.347179 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:16:57.348189 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:16:57.348167 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:17:00.535062 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.535025 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh"] Jul 14 12:17:00.535452 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.535386 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" Jul 14 12:17:00.535452 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.535399 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" Jul 14 12:17:00.535452 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.535423 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" Jul 14 12:17:00.535452 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.535429 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" Jul 14 12:17:00.535452 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.535435 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" Jul 14 12:17:00.535452 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.535441 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e72a415-e4b1-4078-bcab-efa8d4d645a5" containerName="istio-proxy" Jul 14 12:17:00.535651 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.535499 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" Jul 14 12:17:00.535651 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.535510 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0abe9e-6b15-472a-828d-af5eb2ca66b4" containerName="istio-proxy" Jul 14 12:17:00.537840 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.537824 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" Jul 14 12:17:00.541952 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.541932 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:17:00.542102 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.541968 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:17:00.550184 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.550161 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh"] Jul 14 12:17:00.638626 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.638595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae898be-c0b3-46d7-b8eb-aba6efbf8e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh\" (UID: \"5ae898be-c0b3-46d7-b8eb-aba6efbf8e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" Jul 14 12:17:00.739474 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.739436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae898be-c0b3-46d7-b8eb-aba6efbf8e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh\" (UID: \"5ae898be-c0b3-46d7-b8eb-aba6efbf8e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" Jul 14 12:17:00.740032 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.740013 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae898be-c0b3-46d7-b8eb-aba6efbf8e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh\" (UID: \"5ae898be-c0b3-46d7-b8eb-aba6efbf8e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" Jul 14 12:17:00.848073 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.847993 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" Jul 14 12:17:00.985711 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:00.985686 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh"] Jul 14 12:17:00.987466 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:17:00.987432 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae898be_c0b3_46d7_b8eb_aba6efbf8e10.slice/crio-4cb7c63ed33783a31feff8fc2a45be523e0cc0ed2585534bf7d7d0db4f0749ce WatchSource:0}: Error finding container 4cb7c63ed33783a31feff8fc2a45be523e0cc0ed2585534bf7d7d0db4f0749ce: Status 404 returned error can't find the container with id 4cb7c63ed33783a31feff8fc2a45be523e0cc0ed2585534bf7d7d0db4f0749ce Jul 14 12:17:01.753939 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:01.753900 2562 generic.go:356] "Generic (PLEG): container finished" podID="5ae898be-c0b3-46d7-b8eb-aba6efbf8e10" containerID="c0dc05b7e744158001a55e3ce79282c9f0b4ef9c633a540fe89a1a15c7126d2f" exitCode=0 Jul 14 12:17:01.754411 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:01.753958 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" event={"ID":"5ae898be-c0b3-46d7-b8eb-aba6efbf8e10","Type":"ContainerDied","Data":"c0dc05b7e744158001a55e3ce79282c9f0b4ef9c633a540fe89a1a15c7126d2f"} Jul 14 12:17:01.754411 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:01.754012 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" event={"ID":"5ae898be-c0b3-46d7-b8eb-aba6efbf8e10","Type":"ContainerStarted","Data":"4cb7c63ed33783a31feff8fc2a45be523e0cc0ed2585534bf7d7d0db4f0749ce"} Jul 14 12:17:02.892757 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:02.892730 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" Jul 14 12:17:02.960652 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:02.960620 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae898be-c0b3-46d7-b8eb-aba6efbf8e10-maas-api-service-ca\") pod \"5ae898be-c0b3-46d7-b8eb-aba6efbf8e10\" (UID: \"5ae898be-c0b3-46d7-b8eb-aba6efbf8e10\") " Jul 14 12:17:02.960982 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:02.960960 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae898be-c0b3-46d7-b8eb-aba6efbf8e10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ae898be-c0b3-46d7-b8eb-aba6efbf8e10" (UID: "5ae898be-c0b3-46d7-b8eb-aba6efbf8e10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:17:03.062178 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:03.062058 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae898be-c0b3-46d7-b8eb-aba6efbf8e10-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:03.765731 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:03.765638 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" Jul 14 12:17:03.765731 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:03.765682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh" event={"ID":"5ae898be-c0b3-46d7-b8eb-aba6efbf8e10","Type":"ContainerDied","Data":"4cb7c63ed33783a31feff8fc2a45be523e0cc0ed2585534bf7d7d0db4f0749ce"} Jul 14 12:17:03.765933 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:03.765740 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cb7c63ed33783a31feff8fc2a45be523e0cc0ed2585534bf7d7d0db4f0749ce" Jul 14 12:17:26.368266 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.367462 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls"] Jul 14 12:17:26.368266 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.367716 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" containerID="cri-o://3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d" gracePeriod=30 Jul 14 12:17:26.744534 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.744488 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t"] Jul 14 12:17:26.744859 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.744847 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae898be-c0b3-46d7-b8eb-aba6efbf8e10" containerName="revoke-keys" Jul 14 12:17:26.744906 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.744860 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae898be-c0b3-46d7-b8eb-aba6efbf8e10" containerName="revoke-keys" Jul 14 12:17:26.744952 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.744925 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae898be-c0b3-46d7-b8eb-aba6efbf8e10" containerName="revoke-keys" Jul 14 12:17:26.747929 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.747913 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" Jul 14 12:17:26.750850 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.750830 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7pf27\"" Jul 14 12:17:26.750953 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.750833 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 12:17:26.763420 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.763396 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t"] Jul 14 12:17:26.873858 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.873819 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39076dc-fe92-4f74-8785-c24364d24766-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-808361-r789t\" (UID: \"b39076dc-fe92-4f74-8785-c24364d24766\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" Jul 14 12:17:26.975223 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.975185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39076dc-fe92-4f74-8785-c24364d24766-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-808361-r789t\" (UID: \"b39076dc-fe92-4f74-8785-c24364d24766\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" Jul 14 12:17:26.975815 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:26.975795 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39076dc-fe92-4f74-8785-c24364d24766-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-808361-r789t\" (UID: \"b39076dc-fe92-4f74-8785-c24364d24766\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" Jul 14 12:17:27.058469 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:27.058376 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" Jul 14 12:17:27.198374 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:27.198347 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t"] Jul 14 12:17:27.200514 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:17:27.200492 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39076dc_fe92_4f74_8785_c24364d24766.slice/crio-4838affff69e8232551c583fc756e95d43531d93e25fc97e92315ed2c44aa976 WatchSource:0}: Error finding container 4838affff69e8232551c583fc756e95d43531d93e25fc97e92315ed2c44aa976: Status 404 returned error can't find the container with id 4838affff69e8232551c583fc756e95d43531d93e25fc97e92315ed2c44aa976 Jul 14 12:17:27.350346 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:27.350307 2562 patch_prober.go:28] interesting pod/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 12:17:27.350469 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:27.350376 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 12:17:27.898057 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:27.898018 2562 generic.go:356] "Generic (PLEG): container finished" podID="b39076dc-fe92-4f74-8785-c24364d24766" containerID="94c1872b18c61ab00e9759637438b3c35f87993c9d4dedad51ea42927dcdfdf6" exitCode=0 Jul 14 12:17:27.898486 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:27.898111 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" event={"ID":"b39076dc-fe92-4f74-8785-c24364d24766","Type":"ContainerDied","Data":"94c1872b18c61ab00e9759637438b3c35f87993c9d4dedad51ea42927dcdfdf6"} Jul 14 12:17:27.898486 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:27.898180 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" event={"ID":"b39076dc-fe92-4f74-8785-c24364d24766","Type":"ContainerStarted","Data":"4838affff69e8232551c583fc756e95d43531d93e25fc97e92315ed2c44aa976"} Jul 14 12:17:29.049263 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:29.049235 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" Jul 14 12:17:29.196045 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:29.195950 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39076dc-fe92-4f74-8785-c24364d24766-maas-api-service-ca\") pod \"b39076dc-fe92-4f74-8785-c24364d24766\" (UID: \"b39076dc-fe92-4f74-8785-c24364d24766\") " Jul 14 12:17:29.196364 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:29.196342 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39076dc-fe92-4f74-8785-c24364d24766-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b39076dc-fe92-4f74-8785-c24364d24766" (UID: "b39076dc-fe92-4f74-8785-c24364d24766"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:17:29.297599 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:29.297559 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39076dc-fe92-4f74-8785-c24364d24766-maas-api-service-ca\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:29.909663 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:29.909579 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" Jul 14 12:17:29.909663 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:29.909623 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-808361-r789t" event={"ID":"b39076dc-fe92-4f74-8785-c24364d24766","Type":"ContainerDied","Data":"4838affff69e8232551c583fc756e95d43531d93e25fc97e92315ed2c44aa976"} Jul 14 12:17:29.909846 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:29.909669 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4838affff69e8232551c583fc756e95d43531d93e25fc97e92315ed2c44aa976" Jul 14 12:17:31.611805 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.611779 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:17:31.617402 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617381 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-socket\") pod \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " Jul 14 12:17:31.617402 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617409 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-data\") pod \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " Jul 14 12:17:31.617579 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617429 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-certs\") pod \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " Jul 14 12:17:31.617579 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617453 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-podinfo\") pod \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " Jul 14 12:17:31.617579 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617481 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-credential-socket\") pod \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " Jul 14 12:17:31.617579 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617509 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istiod-ca-cert\") pod \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " Jul 14 12:17:31.617798 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617581 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh4sb\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-kube-api-access-nh4sb\") pod \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " Jul 14 12:17:31.617798 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617622 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-token\") pod \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " Jul 14 12:17:31.617798 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617633 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "131bd90b-6f48-4fa8-8e06-067eb7cfc683" (UID: "131bd90b-6f48-4fa8-8e06-067eb7cfc683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:17:31.617798 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617656 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-envoy\") pod \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\" (UID: \"131bd90b-6f48-4fa8-8e06-067eb7cfc683\") " Jul 14 12:17:31.617798 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617709 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "131bd90b-6f48-4fa8-8e06-067eb7cfc683" (UID: "131bd90b-6f48-4fa8-8e06-067eb7cfc683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:17:31.618073 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617967 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:31.618073 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617967 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "131bd90b-6f48-4fa8-8e06-067eb7cfc683" (UID: "131bd90b-6f48-4fa8-8e06-067eb7cfc683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:17:31.618073 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.617984 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:31.618258 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.618146 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "131bd90b-6f48-4fa8-8e06-067eb7cfc683" (UID: "131bd90b-6f48-4fa8-8e06-067eb7cfc683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:17:31.618378 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.618337 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-data" (OuterVolumeSpecName: "istio-data") pod "131bd90b-6f48-4fa8-8e06-067eb7cfc683" (UID: "131bd90b-6f48-4fa8-8e06-067eb7cfc683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:17:31.619868 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.619837 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "131bd90b-6f48-4fa8-8e06-067eb7cfc683" (UID: "131bd90b-6f48-4fa8-8e06-067eb7cfc683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:17:31.619868 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.619839 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-kube-api-access-nh4sb" (OuterVolumeSpecName: "kube-api-access-nh4sb") pod "131bd90b-6f48-4fa8-8e06-067eb7cfc683" (UID: "131bd90b-6f48-4fa8-8e06-067eb7cfc683"). InnerVolumeSpecName "kube-api-access-nh4sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:17:31.620223 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.620203 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-token" (OuterVolumeSpecName: "istio-token") pod "131bd90b-6f48-4fa8-8e06-067eb7cfc683" (UID: "131bd90b-6f48-4fa8-8e06-067eb7cfc683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:17:31.620308 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.620235 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "131bd90b-6f48-4fa8-8e06-067eb7cfc683" (UID: "131bd90b-6f48-4fa8-8e06-067eb7cfc683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:17:31.719146 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.719038 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh4sb\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-kube-api-access-nh4sb\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:31.719146 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.719069 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:31.719146 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.719098 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:31.719146 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.719107 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:31.719146 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.719114 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:31.719146 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.719122 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bd90b-6f48-4fa8-8e06-067eb7cfc683-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:31.719146 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.719130 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bd90b-6f48-4fa8-8e06-067eb7cfc683-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:17:31.922856 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.922821 2562 generic.go:356] "Generic (PLEG): container finished" podID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerID="3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d" exitCode=0 Jul 14 12:17:31.923040 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.922870 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" Jul 14 12:17:31.923040 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.922894 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" event={"ID":"131bd90b-6f48-4fa8-8e06-067eb7cfc683","Type":"ContainerDied","Data":"3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d"} Jul 14 12:17:31.923040 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.922938 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls" event={"ID":"131bd90b-6f48-4fa8-8e06-067eb7cfc683","Type":"ContainerDied","Data":"165b76b9de914e023953102a2dc05d11053ecb5c4419cebd7a1dc322c23b7fd1"} Jul 14 12:17:31.923040 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.922959 2562 scope.go:117] "RemoveContainer" containerID="3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d" Jul 14 12:17:31.933159 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.933121 2562 scope.go:117] "RemoveContainer" containerID="b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584" Jul 14 12:17:31.942312 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.942288 2562 scope.go:117] "RemoveContainer" containerID="3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d" Jul 14 12:17:31.942597 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:17:31.942576 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d\": container with ID starting with 3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d not found: ID does not exist" containerID="3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d" Jul 14 12:17:31.942669 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.942607 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d"} err="failed to get container status \"3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d\": rpc error: code = NotFound desc = could not find container \"3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d\": container with ID starting with 3c627d3b4db67c67fe898de17dbd29dafc05b706bfe61a99c0d61a2d0772079d not found: ID does not exist" Jul 14 12:17:31.942669 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.942626 2562 scope.go:117] "RemoveContainer" containerID="b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584" Jul 14 12:17:31.942876 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:17:31.942854 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584\": container with ID starting with b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584 not found: ID does not exist" containerID="b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584" Jul 14 12:17:31.942922 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.942884 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584"} err="failed to get container status \"b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584\": rpc error: code = NotFound desc = could not find container \"b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584\": container with ID starting with b8bdbba1670d60f6dae6ff974c31629154bc1f930bc4eeeaedb39f43438f1584 not found: ID does not exist" Jul 14 12:17:31.952327 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.952298 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls"] Jul 14 12:17:31.957350 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:31.957325 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3ce01d-openshift-default-7c6f55c48c-m7sls"] Jul 14 12:17:33.478997 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:33.478962 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" path="/var/lib/kubelet/pods/131bd90b-6f48-4fa8-8e06-067eb7cfc683/volumes" Jul 14 12:17:47.539241 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:47.539212 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:17:47.543033 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:47.543005 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:17:47.549813 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:47.549792 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:17:47.553197 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:47.553178 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:17:53.653834 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.653798 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6"] Jul 14 12:17:53.654321 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.654240 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b39076dc-fe92-4f74-8785-c24364d24766" containerName="revoke-keys" Jul 14 12:17:53.654321 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.654254 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b39076dc-fe92-4f74-8785-c24364d24766" containerName="revoke-keys" Jul 14 12:17:53.654321 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.654266 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" Jul 14 12:17:53.654321 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.654272 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" Jul 14 12:17:53.654321 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.654296 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" Jul 14 12:17:53.654321 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.654303 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" Jul 14 12:17:53.654549 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.654366 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" Jul 14 12:17:53.654549 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.654376 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b39076dc-fe92-4f74-8785-c24364d24766" containerName="revoke-keys" Jul 14 12:17:53.654549 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.654382 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="131bd90b-6f48-4fa8-8e06-067eb7cfc683" containerName="istio-proxy" Jul 14 12:17:53.657508 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.657490 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.660795 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.660772 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-cac82f-openshift-default-dockercfg-2s9tq\"" Jul 14 12:17:53.673606 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.673577 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6"] Jul 14 12:17:53.718645 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.718603 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-certs\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.718822 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.718663 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-socket\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.718822 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.718694 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-token\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.718822 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.718761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-credential-socket\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.718822 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.718801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-podinfo\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.718965 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.718837 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9df8\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-kube-api-access-d9df8\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.718965 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.718859 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-data\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.718965 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.718874 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f62080-dd96-4664-90fd-b2e3e9cb6879-istiod-ca-cert\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.718965 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.718908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-envoy\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820329 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820283 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9df8\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-kube-api-access-d9df8\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820544 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-data\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820544 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f62080-dd96-4664-90fd-b2e3e9cb6879-istiod-ca-cert\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820544 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-envoy\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820544 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820456 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-certs\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820544 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-socket\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820544 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-token\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820873 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-credential-socket\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820873 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-podinfo\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.820873 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.820825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-data\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.821153 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.821127 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-socket\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.821243 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.821173 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-certs\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.821243 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.821219 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f62080-dd96-4664-90fd-b2e3e9cb6879-istiod-ca-cert\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.821361 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.821308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-credential-socket\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.822839 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.822818 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-envoy\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.823056 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.823038 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-podinfo\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.831487 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.831463 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-token\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.831621 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.831602 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9df8\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-kube-api-access-d9df8\") pod \"e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:53.968510 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:53.968418 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:54.120996 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:54.120970 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6"] Jul 14 12:17:54.122676 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:17:54.122647 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f62080_dd96_4664_90fd_b2e3e9cb6879.slice/crio-d6382acfaeb461457dc7389aacece7c0d440081224a3c84937a8471d23e3d820 WatchSource:0}: Error finding container d6382acfaeb461457dc7389aacece7c0d440081224a3c84937a8471d23e3d820: Status 404 returned error can't find the container with id d6382acfaeb461457dc7389aacece7c0d440081224a3c84937a8471d23e3d820 Jul 14 12:17:54.124755 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:54.124720 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:17:54.124832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:54.124790 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:17:54.124832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:54.124819 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:17:55.062905 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:55.062866 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" event={"ID":"98f62080-dd96-4664-90fd-b2e3e9cb6879","Type":"ContainerStarted","Data":"b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e"} Jul 14 12:17:55.063350 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:55.062955 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" event={"ID":"98f62080-dd96-4664-90fd-b2e3e9cb6879","Type":"ContainerStarted","Data":"d6382acfaeb461457dc7389aacece7c0d440081224a3c84937a8471d23e3d820"} Jul 14 12:17:55.089066 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:55.089019 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" podStartSLOduration=2.089003652 podStartE2EDuration="2.089003652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:17:55.087913125 +0000 UTC m=+2408.163767739" watchObservedRunningTime="2026-07-14 12:17:55.089003652 +0000 UTC m=+2408.164858241" Jul 14 12:17:55.968910 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:55.968865 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:55.973951 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:55.973923 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:56.068339 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:56.068310 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:17:56.069384 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:17:56.069364 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:18:02.204729 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.204698 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q"] Jul 14 12:18:02.208352 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.208335 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.213784 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.213765 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-480a7f-openshift-default-dockercfg-vmdbk\"" Jul 14 12:18:02.237780 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.237753 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q"] Jul 14 12:18:02.300087 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.300050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-podinfo\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.300295 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.300182 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-socket\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.300295 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.300229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-certs\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.300425 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.300302 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-envoy\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.300425 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.300373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-credential-socket\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.300509 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.300441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-token\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.300509 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.300490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslwr\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-kube-api-access-xslwr\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.300590 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.300528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istiod-ca-cert\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.300627 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.300611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-data\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.401603 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.401547 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-socket\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.401603 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.401609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-certs\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.401877 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.401647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-envoy\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.401877 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.401702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-credential-socket\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.401877 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.401742 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-token\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.401877 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.401766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslwr\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-kube-api-access-xslwr\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.401877 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.401795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istiod-ca-cert\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.401877 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.401870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-data\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.402223 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.401923 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-podinfo\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.402223 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.402017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-socket\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.402684 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.402655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-credential-socket\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.402826 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.402671 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-certs\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.403229 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.403205 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-data\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.403617 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.403597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istiod-ca-cert\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.404666 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.404644 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-podinfo\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.404841 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.404824 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-envoy\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.414405 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.414383 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-token\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.416990 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.416962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslwr\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-kube-api-access-xslwr\") pod \"e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.518873 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.518840 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:02.726528 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.726499 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q"] Jul 14 12:18:02.727425 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:18:02.727399 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e163dc6_ed1a_48ae_b12e_ff4eed049d1d.slice/crio-638894838dde0d133c81bd9b8a55998f27afbe4d1540ba310975ba46f48833a2 WatchSource:0}: Error finding container 638894838dde0d133c81bd9b8a55998f27afbe4d1540ba310975ba46f48833a2: Status 404 returned error can't find the container with id 638894838dde0d133c81bd9b8a55998f27afbe4d1540ba310975ba46f48833a2 Jul 14 12:18:02.729399 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.729366 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:18:02.729509 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.729444 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:18:02.729509 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:02.729490 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:18:03.109148 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:03.109036 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" event={"ID":"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d","Type":"ContainerStarted","Data":"6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05"} Jul 14 12:18:03.109148 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:03.109102 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" event={"ID":"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d","Type":"ContainerStarted","Data":"638894838dde0d133c81bd9b8a55998f27afbe4d1540ba310975ba46f48833a2"} Jul 14 12:18:03.144940 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:03.144854 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" podStartSLOduration=1.144830377 podStartE2EDuration="1.144830377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 12:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:18:03.140028063 +0000 UTC m=+2416.215882654" watchObservedRunningTime="2026-07-14 12:18:03.144830377 +0000 UTC m=+2416.220684966" Jul 14 12:18:03.519259 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:03.519224 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:03.524306 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:03.524276 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:04.122604 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:04.122564 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:04.123674 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:04.123654 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:10.164399 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:10.164359 2562 generic.go:356] "Generic (PLEG): container finished" podID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerID="b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e" exitCode=0 Jul 14 12:18:10.164878 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:10.164428 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" event={"ID":"98f62080-dd96-4664-90fd-b2e3e9cb6879","Type":"ContainerDied","Data":"b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e"} Jul 14 12:18:10.164878 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:10.164850 2562 scope.go:117] "RemoveContainer" containerID="b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e" Jul 14 12:18:10.165842 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:10.165812 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:18:10.165897 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:10.165884 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:18:10.165937 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:10.165920 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:18:11.068629 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:11.068584 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:18:11.176071 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:11.176025 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" event={"ID":"98f62080-dd96-4664-90fd-b2e3e9cb6879","Type":"ContainerStarted","Data":"5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5"} Jul 14 12:18:11.969246 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:11.969210 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:18:11.970574 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:11.970535 2562 patch_prober.go:28] interesting pod/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 14 12:18:11.970725 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:11.970622 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 14 12:18:12.969589 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:12.969535 2562 patch_prober.go:28] interesting pod/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 14 12:18:12.970150 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:12.969613 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 14 12:18:13.969186 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:13.969139 2562 patch_prober.go:28] interesting pod/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 14 12:18:13.969416 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:13.969213 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 14 12:18:14.969162 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:14.969123 2562 patch_prober.go:28] interesting pod/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 14 12:18:14.969570 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:14.969181 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 14 12:18:15.972922 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:15.972887 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:18:15.973423 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:15.973291 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:18:15.973992 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:15.973971 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:18:18.220997 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:18.220963 2562 generic.go:356] "Generic (PLEG): container finished" podID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerID="6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05" exitCode=0 Jul 14 12:18:18.221450 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:18.221047 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" event={"ID":"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d","Type":"ContainerDied","Data":"6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05"} Jul 14 12:18:18.221494 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:18.221468 2562 scope.go:117] "RemoveContainer" containerID="6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05" Jul 14 12:18:18.222590 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:18.222561 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:18:18.222655 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:18.222629 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:18:18.222701 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:18.222666 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 12:18:19.123588 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:19.123551 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:19.228530 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:19.228493 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" event={"ID":"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d","Type":"ContainerStarted","Data":"7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e"} Jul 14 12:18:19.519804 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:19.519768 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:19.520950 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:19.520923 2562 patch_prober.go:28] interesting pod/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 14 12:18:19.521063 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:19.520974 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 14 12:18:20.519605 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:20.519567 2562 patch_prober.go:28] interesting pod/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 14 12:18:20.520017 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:20.519627 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 14 12:18:21.519875 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:21.519841 2562 patch_prober.go:28] interesting pod/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 14 12:18:21.520306 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:21.519892 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 14 12:18:22.519709 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:22.519668 2562 patch_prober.go:28] interesting pod/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 14 12:18:22.519909 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:22.519730 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 14 12:18:23.520298 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:23.520260 2562 patch_prober.go:28] interesting pod/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 14 12:18:23.520686 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:23.520322 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 14 12:18:24.523686 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:24.523652 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:24.524291 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:24.524037 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:18:24.524832 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:18:24.524814 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:21:47.708745 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:21:47.708714 2562 scope.go:117] "RemoveContainer" containerID="e2196c88055365fa41866ffb9fc219e32ca411dd46d813229ea69e131ab8c9ff" Jul 14 12:22:47.578810 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:22:47.578742 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:22:47.582374 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:22:47.582353 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:22:47.587993 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:22:47.587976 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:22:47.591461 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:22:47.591444 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-md69v_f7eac93e-56a6-4d90-aa2e-507659c30532/ovn-acl-logging/0.log" Jul 14 12:24:13.278822 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:13.278738 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6"] Jul 14 12:24:13.279309 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:13.278949 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" containerID="cri-o://5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5" gracePeriod=30 Jul 14 12:24:15.975202 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:15.975166 2562 patch_prober.go:28] interesting pod/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 12:24:15.975560 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:15.975229 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 12:24:18.542657 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.542600 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:24:18.585045 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585016 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-data\") pod \"98f62080-dd96-4664-90fd-b2e3e9cb6879\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " Jul 14 12:24:18.585216 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585056 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9df8\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-kube-api-access-d9df8\") pod \"98f62080-dd96-4664-90fd-b2e3e9cb6879\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " Jul 14 12:24:18.585216 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585111 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-envoy\") pod \"98f62080-dd96-4664-90fd-b2e3e9cb6879\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " Jul 14 12:24:18.585216 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585140 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f62080-dd96-4664-90fd-b2e3e9cb6879-istiod-ca-cert\") pod \"98f62080-dd96-4664-90fd-b2e3e9cb6879\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " Jul 14 12:24:18.585216 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585193 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-socket\") pod \"98f62080-dd96-4664-90fd-b2e3e9cb6879\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " Jul 14 12:24:18.585422 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585229 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-podinfo\") pod \"98f62080-dd96-4664-90fd-b2e3e9cb6879\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " Jul 14 12:24:18.585422 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585257 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-certs\") pod \"98f62080-dd96-4664-90fd-b2e3e9cb6879\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " Jul 14 12:24:18.585422 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585285 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-token\") pod \"98f62080-dd96-4664-90fd-b2e3e9cb6879\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " Jul 14 12:24:18.585422 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585339 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-credential-socket\") pod \"98f62080-dd96-4664-90fd-b2e3e9cb6879\" (UID: \"98f62080-dd96-4664-90fd-b2e3e9cb6879\") " Jul 14 12:24:18.585684 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585610 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "98f62080-dd96-4664-90fd-b2e3e9cb6879" (UID: "98f62080-dd96-4664-90fd-b2e3e9cb6879"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:18.585684 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585567 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "98f62080-dd96-4664-90fd-b2e3e9cb6879" (UID: "98f62080-dd96-4664-90fd-b2e3e9cb6879"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:18.585684 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585617 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f62080-dd96-4664-90fd-b2e3e9cb6879-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "98f62080-dd96-4664-90fd-b2e3e9cb6879" (UID: "98f62080-dd96-4664-90fd-b2e3e9cb6879"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:24:18.585964 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585733 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:18.585964 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585756 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:18.585964 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.585796 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "98f62080-dd96-4664-90fd-b2e3e9cb6879" (UID: "98f62080-dd96-4664-90fd-b2e3e9cb6879"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:18.586161 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.586030 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-data" (OuterVolumeSpecName: "istio-data") pod "98f62080-dd96-4664-90fd-b2e3e9cb6879" (UID: "98f62080-dd96-4664-90fd-b2e3e9cb6879"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:18.587387 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.587365 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "98f62080-dd96-4664-90fd-b2e3e9cb6879" (UID: "98f62080-dd96-4664-90fd-b2e3e9cb6879"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:18.587495 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.587475 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-token" (OuterVolumeSpecName: "istio-token") pod "98f62080-dd96-4664-90fd-b2e3e9cb6879" (UID: "98f62080-dd96-4664-90fd-b2e3e9cb6879"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:24:18.587706 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.587684 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-kube-api-access-d9df8" (OuterVolumeSpecName: "kube-api-access-d9df8") pod "98f62080-dd96-4664-90fd-b2e3e9cb6879" (UID: "98f62080-dd96-4664-90fd-b2e3e9cb6879"). InnerVolumeSpecName "kube-api-access-d9df8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:24:18.588070 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.588054 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "98f62080-dd96-4664-90fd-b2e3e9cb6879" (UID: "98f62080-dd96-4664-90fd-b2e3e9cb6879"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:24:18.687049 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.687020 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:18.687049 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.687048 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:18.687233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.687058 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9df8\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-kube-api-access-d9df8\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:18.687233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.687069 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:18.687233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.687099 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f62080-dd96-4664-90fd-b2e3e9cb6879-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:18.687233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.687107 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:18.687233 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:18.687114 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f62080-dd96-4664-90fd-b2e3e9cb6879-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:19.301631 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.301600 2562 generic.go:356] "Generic (PLEG): container finished" podID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerID="5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5" exitCode=0 Jul 14 12:24:19.301805 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.301641 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" event={"ID":"98f62080-dd96-4664-90fd-b2e3e9cb6879","Type":"ContainerDied","Data":"5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5"} Jul 14 12:24:19.301805 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.301648 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" Jul 14 12:24:19.301805 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.301690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6" event={"ID":"98f62080-dd96-4664-90fd-b2e3e9cb6879","Type":"ContainerDied","Data":"d6382acfaeb461457dc7389aacece7c0d440081224a3c84937a8471d23e3d820"} Jul 14 12:24:19.301805 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.301716 2562 scope.go:117] "RemoveContainer" containerID="5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5" Jul 14 12:24:19.311326 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.311308 2562 scope.go:117] "RemoveContainer" containerID="b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e" Jul 14 12:24:19.319368 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.319302 2562 scope.go:117] "RemoveContainer" containerID="5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5" Jul 14 12:24:19.319544 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:24:19.319527 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5\": container with ID starting with 5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5 not found: ID does not exist" containerID="5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5" Jul 14 12:24:19.319589 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.319551 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5"} err="failed to get container status \"5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5\": rpc error: code = NotFound desc = could not find container \"5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5\": container with ID starting with 5f75ac3f32898e61670120164bb4b279ce05637d3649c10c4b27fa9e8b3d51f5 not found: ID does not exist" Jul 14 12:24:19.319589 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.319566 2562 scope.go:117] "RemoveContainer" containerID="b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e" Jul 14 12:24:19.319767 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:24:19.319749 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e\": container with ID starting with b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e not found: ID does not exist" containerID="b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e" Jul 14 12:24:19.319810 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.319773 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e"} err="failed to get container status \"b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e\": rpc error: code = NotFound desc = could not find container \"b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e\": container with ID starting with b51cce3db93b93dc7289092715b2f71b50a0d0c784c5c36ccd1558ab58a7030e not found: ID does not exist" Jul 14 12:24:19.329289 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.329266 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6"] Jul 14 12:24:19.336243 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.336222 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-cac82f-openshift-default-579d9df5dc-nxls6"] Jul 14 12:24:19.486524 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:19.486493 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" path="/var/lib/kubelet/pods/98f62080-dd96-4664-90fd-b2e3e9cb6879/volumes" Jul 14 12:24:39.542069 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:39.542034 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q"] Jul 14 12:24:39.542658 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:39.542324 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" containerID="cri-o://7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e" gracePeriod=30 Jul 14 12:24:44.799037 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.799012 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:24:44.899787 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.899711 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istiod-ca-cert\") pod \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " Jul 14 12:24:44.899787 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.899761 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-podinfo\") pod \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " Jul 14 12:24:44.899957 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.899800 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xslwr\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-kube-api-access-xslwr\") pod \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " Jul 14 12:24:44.899957 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.899827 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-data\") pod \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " Jul 14 12:24:44.899957 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.899910 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-envoy\") pod \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " Jul 14 12:24:44.899957 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.899938 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-token\") pod \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " Jul 14 12:24:44.900195 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.899965 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-socket\") pod \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " Jul 14 12:24:44.900195 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.899993 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-credential-socket\") pod \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " Jul 14 12:24:44.900195 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.900021 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-certs\") pod \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\" (UID: \"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d\") " Jul 14 12:24:44.900195 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.900045 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" (UID: "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 12:24:44.900417 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.900243 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" (UID: "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:44.900417 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.900324 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:44.900417 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.900344 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istiod-ca-cert\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:44.900417 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.900319 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" (UID: "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:44.900417 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.900378 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" (UID: "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:44.900710 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.900686 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" (UID: "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:44.902030 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.902006 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-kube-api-access-xslwr" (OuterVolumeSpecName: "kube-api-access-xslwr") pod "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" (UID: "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d"). InnerVolumeSpecName "kube-api-access-xslwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:24:44.902205 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.902157 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" (UID: "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 12:24:44.902433 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.902404 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" (UID: "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 12:24:44.902523 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:44.902471 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" (UID: "9e163dc6-ed1a-48ae-b12e-ff4eed049d1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 12:24:45.001063 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.001039 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xslwr\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-kube-api-access-xslwr\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:45.001063 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.001061 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-data\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:45.001252 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.001071 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-envoy\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:45.001252 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.001095 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-token\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:45.001252 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.001103 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-credential-socket\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:45.001252 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.001111 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-workload-certs\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:45.001252 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.001120 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d-istio-podinfo\") on node \"ip-10-0-132-116.ec2.internal\" DevicePath \"\"" Jul 14 12:24:45.451658 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.451627 2562 generic.go:356] "Generic (PLEG): container finished" podID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerID="7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e" exitCode=0 Jul 14 12:24:45.451833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.451673 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" Jul 14 12:24:45.451833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.451690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" event={"ID":"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d","Type":"ContainerDied","Data":"7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e"} Jul 14 12:24:45.451833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.451733 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q" event={"ID":"9e163dc6-ed1a-48ae-b12e-ff4eed049d1d","Type":"ContainerDied","Data":"638894838dde0d133c81bd9b8a55998f27afbe4d1540ba310975ba46f48833a2"} Jul 14 12:24:45.451833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.451753 2562 scope.go:117] "RemoveContainer" containerID="7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e" Jul 14 12:24:45.462108 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.462066 2562 scope.go:117] "RemoveContainer" containerID="6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05" Jul 14 12:24:45.470806 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.470789 2562 scope.go:117] "RemoveContainer" containerID="7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e" Jul 14 12:24:45.471042 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:24:45.471026 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e\": container with ID starting with 7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e not found: ID does not exist" containerID="7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e" Jul 14 12:24:45.471138 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.471049 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e"} err="failed to get container status \"7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e\": rpc error: code = NotFound desc = could not find container \"7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e\": container with ID starting with 7956a91ef65be3904823996d88695e6464f49f582879ca903f45a691e1f3ae9e not found: ID does not exist" Jul 14 12:24:45.471138 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.471064 2562 scope.go:117] "RemoveContainer" containerID="6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05" Jul 14 12:24:45.471358 ip-10-0-132-116 kubenswrapper[2562]: E0714 12:24:45.471337 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05\": container with ID starting with 6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05 not found: ID does not exist" containerID="6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05" Jul 14 12:24:45.471407 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.471363 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05"} err="failed to get container status \"6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05\": rpc error: code = NotFound desc = could not find container \"6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05\": container with ID starting with 6652f9de96af4233e9ec93caf7de4c3015ee696898972935f7026c5d2f495d05 not found: ID does not exist" Jul 14 12:24:45.479537 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.479514 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q"] Jul 14 12:24:45.485283 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.485264 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-480a7f-openshift-default-68b4fd5555-d6m2q"] Jul 14 12:24:45.924812 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.924787 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c944d8cbf-f4dct_4bc000c4-a69a-4777-90f5-36ec02e03702/copy-manifests/0.log" Jul 14 12:24:45.933901 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:45.933873 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c944d8cbf-f4dct_4bc000c4-a69a-4777-90f5-36ec02e03702/manager/0.log" Jul 14 12:24:46.675425 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:46.675393 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733840-ftwtz_ecc0ee43-8711-4276-9c3a-e950b55a1e6e/cleanup/0.log" Jul 14 12:24:46.906303 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:46.906268 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-0135d283-pfdgd_571260b0-aa8a-41c8-860f-387924e08fc9/revoke-keys/0.log" Jul 14 12:24:47.020824 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:47.020798 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-ab110be5-kkpw7_93ab358c-5910-40c3-95b4-552358130286/revoke-keys/0.log" Jul 14 12:24:47.133103 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:47.133040 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-3ce01d-mdwfh_5ae898be-c0b3-46d7-b8eb-aba6efbf8e10/revoke-keys/0.log" Jul 14 12:24:47.242826 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:47.242800 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-808361-r789t_b39076dc-fe92-4f74-8785-c24364d24766/revoke-keys/0.log" Jul 14 12:24:47.368341 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:47.368271 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-91627dc1-8vq4h_6ab97a9c-a1ef-495e-bf1c-42de59162ed6/revoke-keys/0.log" Jul 14 12:24:47.486758 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:47.486727 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" path="/var/lib/kubelet/pods/9e163dc6-ed1a-48ae-b12e-ff4eed049d1d/volumes" Jul 14 12:24:47.610417 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:47.610385 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-a015ba-vz7zp_752b16b7-9a6e-400d-9e46-009eb07e670d/revoke-keys/0.log" Jul 14 12:24:47.722821 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:47.722801 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-050e1b21-4kr4p_95aebf37-854e-49c7-b78b-6a98b0879bed/revoke-keys/0.log" Jul 14 12:24:47.837435 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:47.837404 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-5374a9b0-mw6vd_f4a2867e-2956-4159-8fdc-1e2ab336a5c8/revoke-keys/0.log" Jul 14 12:24:47.950181 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:47.950149 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-c52b78c6-pzdgj_35ab9b59-4bab-459e-9f5f-1cc5f146c60e/revoke-keys/0.log" Jul 14 12:24:49.097773 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.097743 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9_21f46e89-dcda-4bd1-a630-001d517a07a3/util/0.log" Jul 14 12:24:49.106041 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.106022 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9_21f46e89-dcda-4bd1-a630-001d517a07a3/pull/0.log" Jul 14 12:24:49.114188 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.114172 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9_21f46e89-dcda-4bd1-a630-001d517a07a3/extract/0.log" Jul 14 12:24:49.230000 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.229970 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96_5bf9c174-5b84-43bb-b9f5-4c12d25787cd/util/0.log" Jul 14 12:24:49.238311 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.238289 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96_5bf9c174-5b84-43bb-b9f5-4c12d25787cd/pull/0.log" Jul 14 12:24:49.246756 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.246730 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96_5bf9c174-5b84-43bb-b9f5-4c12d25787cd/extract/0.log" Jul 14 12:24:49.363625 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.363549 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg_79b538fa-1efc-4c29-9ea8-504d4ad0bace/pull/0.log" Jul 14 12:24:49.374179 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.374159 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg_79b538fa-1efc-4c29-9ea8-504d4ad0bace/extract/0.log" Jul 14 12:24:49.385670 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.385652 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg_79b538fa-1efc-4c29-9ea8-504d4ad0bace/util/0.log" Jul 14 12:24:49.498049 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.498026 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq_9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9/util/0.log" Jul 14 12:24:49.505387 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.505372 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq_9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9/pull/0.log" Jul 14 12:24:49.513169 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:49.513154 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq_9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9/extract/0.log" Jul 14 12:24:50.838586 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:50.838559 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qkhnv_0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497/istio-proxy/0.log" Jul 14 12:24:50.959996 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:50.959968 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2xhjw_010227ea-94b5-4e23-adce-6efe2c354add/discovery/0.log" Jul 14 12:24:51.446431 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:51.446401 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-tp2ll_aaf74151-7087-4688-b78f-713f36f0a271/payload-pre-processing/0.log" Jul 14 12:24:51.715094 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:51.714988 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7894875846-lcf2d_3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4/router/0.log" Jul 14 12:24:52.265844 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:52.265813 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-chmdd_d57667c5-b5ca-478b-af9c-37ac0365af12/main/0.log" Jul 14 12:24:52.508574 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:24:52.508534 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbknlbm_c6672e86-5394-4557-9520-e179140a3e4c/main/0.log" Jul 14 12:25:00.291410 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:00.291381 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xkhkk_03ec8cdd-c0b1-4e3b-9792-a9bbcdbc75fc/global-pull-secret-syncer/0.log" Jul 14 12:25:00.458020 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:00.457993 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tmggf_55ed8677-a1e3-4247-8b7f-0cca2d6cd535/konnectivity-agent/0.log" Jul 14 12:25:00.480427 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:00.480408 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-116.ec2.internal_ef29f282130d13891da160528c132f5f/haproxy/0.log" Jul 14 12:25:04.901182 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:04.901110 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9_21f46e89-dcda-4bd1-a630-001d517a07a3/extract/0.log" Jul 14 12:25:04.928665 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:04.928642 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9_21f46e89-dcda-4bd1-a630-001d517a07a3/util/0.log" Jul 14 12:25:04.955746 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:04.955723 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b7sf9_21f46e89-dcda-4bd1-a630-001d517a07a3/pull/0.log" Jul 14 12:25:05.000532 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:05.000512 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96_5bf9c174-5b84-43bb-b9f5-4c12d25787cd/extract/0.log" Jul 14 12:25:05.030612 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:05.030592 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96_5bf9c174-5b84-43bb-b9f5-4c12d25787cd/util/0.log" Jul 14 12:25:05.059853 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:05.059831 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kgk96_5bf9c174-5b84-43bb-b9f5-4c12d25787cd/pull/0.log" Jul 14 12:25:05.092694 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:05.092679 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg_79b538fa-1efc-4c29-9ea8-504d4ad0bace/extract/0.log" Jul 14 12:25:05.122761 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:05.122738 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg_79b538fa-1efc-4c29-9ea8-504d4ad0bace/util/0.log" Jul 14 12:25:05.149973 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:05.149954 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n9sxg_79b538fa-1efc-4c29-9ea8-504d4ad0bace/pull/0.log" Jul 14 12:25:05.187996 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:05.187938 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq_9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9/extract/0.log" Jul 14 12:25:05.219635 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:05.219614 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq_9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9/util/0.log" Jul 14 12:25:05.260810 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:05.260788 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dvjrq_9c575d8e-7ae5-4be4-a1e3-af3a6e77aaa9/pull/0.log" Jul 14 12:25:08.077131 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.077100 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ee3ab75-ad74-42f7-a4a0-53e4d126773d/alertmanager/0.log" Jul 14 12:25:08.113265 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.113243 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ee3ab75-ad74-42f7-a4a0-53e4d126773d/config-reloader/0.log" Jul 14 12:25:08.142124 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.142095 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ee3ab75-ad74-42f7-a4a0-53e4d126773d/kube-rbac-proxy-web/0.log" Jul 14 12:25:08.171584 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.171564 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ee3ab75-ad74-42f7-a4a0-53e4d126773d/kube-rbac-proxy/0.log" Jul 14 12:25:08.200341 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.200318 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ee3ab75-ad74-42f7-a4a0-53e4d126773d/kube-rbac-proxy-metric/0.log" Jul 14 12:25:08.226165 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.226144 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ee3ab75-ad74-42f7-a4a0-53e4d126773d/prom-label-proxy/0.log" Jul 14 12:25:08.252535 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.252512 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7ee3ab75-ad74-42f7-a4a0-53e4d126773d/init-config-reloader/0.log" Jul 14 12:25:08.586412 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.586387 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ggjzd_86306431-e7f5-442f-94df-140f5fbdd1d1/node-exporter/0.log" Jul 14 12:25:08.613239 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.613216 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ggjzd_86306431-e7f5-442f-94df-140f5fbdd1d1/kube-rbac-proxy/0.log" Jul 14 12:25:08.637351 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:08.637332 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ggjzd_86306431-e7f5-442f-94df-140f5fbdd1d1/init-textfile/0.log" Jul 14 12:25:09.147946 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:09.147918 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-8dk7w_c8ad185f-7916-4073-baf7-1db23a65afd2/prometheus-operator-admission-webhook/0.log" Jul 14 12:25:10.212208 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212171 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w"] Jul 14 12:25:10.212617 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212590 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" Jul 14 12:25:10.212617 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212612 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" Jul 14 12:25:10.212721 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212635 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" Jul 14 12:25:10.212721 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212644 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" Jul 14 12:25:10.212721 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212662 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" Jul 14 12:25:10.212721 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212671 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" Jul 14 12:25:10.212721 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212680 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" Jul 14 12:25:10.212721 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212685 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" Jul 14 12:25:10.212902 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212743 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e163dc6-ed1a-48ae-b12e-ff4eed049d1d" containerName="istio-proxy" Jul 14 12:25:10.212902 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212752 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" Jul 14 12:25:10.212902 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.212760 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f62080-dd96-4664-90fd-b2e3e9cb6879" containerName="istio-proxy" Jul 14 12:25:10.215844 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.215829 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.219238 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.219188 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6fcf\"/\"kube-root-ca.crt\"" Jul 14 12:25:10.219238 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.219225 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t6fcf\"/\"default-dockercfg-tcbhs\"" Jul 14 12:25:10.220553 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.220530 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6fcf\"/\"openshift-service-ca.crt\"" Jul 14 12:25:10.226353 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.226332 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w"] Jul 14 12:25:10.307655 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.307621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-sys\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.307814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.307679 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-proc\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.307814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.307697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjsbb\" (UniqueName: \"kubernetes.io/projected/e6771c43-4e90-4efe-a66c-777eb1e14845-kube-api-access-gjsbb\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.307814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.307718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-lib-modules\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.307814 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.307789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-podres\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.408733 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.408701 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-proc\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.408733 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.408732 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjsbb\" (UniqueName: \"kubernetes.io/projected/e6771c43-4e90-4efe-a66c-777eb1e14845-kube-api-access-gjsbb\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.408989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.408756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-lib-modules\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.408989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.408777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-podres\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.408989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.408813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-sys\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.408989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.408831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-proc\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.408989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.408891 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-sys\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.408989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.408905 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-lib-modules\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.408989 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.408942 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6771c43-4e90-4efe-a66c-777eb1e14845-podres\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.420369 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.420348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjsbb\" (UniqueName: \"kubernetes.io/projected/e6771c43-4e90-4efe-a66c-777eb1e14845-kube-api-access-gjsbb\") pod \"perf-node-gather-daemonset-rk89w\" (UID: \"e6771c43-4e90-4efe-a66c-777eb1e14845\") " pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.525839 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.525816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:10.658508 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.658482 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-ccmt2_ab73933c-fedc-474f-b0fa-a68e49f0f6ea/networking-console-plugin/0.log" Jul 14 12:25:10.660844 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:10.660802 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w"] Jul 14 12:25:10.661219 ip-10-0-132-116 kubenswrapper[2562]: W0714 12:25:10.661195 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode6771c43_4e90_4efe_a66c_777eb1e14845.slice/crio-8edb82f9b86f9dea7a65219a5741c3ee9abf9c18d47e776b98fcd2db129e835e WatchSource:0}: Error finding container 8edb82f9b86f9dea7a65219a5741c3ee9abf9c18d47e776b98fcd2db129e835e: Status 404 returned error can't find the container with id 8edb82f9b86f9dea7a65219a5741c3ee9abf9c18d47e776b98fcd2db129e835e Jul 14 12:25:11.299027 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:11.298995 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/1.log" Jul 14 12:25:11.303330 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:11.303310 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-k5nq8_a500e755-d871-4992-9b39-18ef1f7e9330/console-operator/2.log" Jul 14 12:25:11.592422 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:11.592335 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" event={"ID":"e6771c43-4e90-4efe-a66c-777eb1e14845","Type":"ContainerStarted","Data":"39eacfed45d5620fca0f819b03d4b97d41d6426ef3a13cbdcbe7270ac7b150e8"} Jul 14 12:25:11.592422 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:11.592383 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" event={"ID":"e6771c43-4e90-4efe-a66c-777eb1e14845","Type":"ContainerStarted","Data":"8edb82f9b86f9dea7a65219a5741c3ee9abf9c18d47e776b98fcd2db129e835e"} Jul 14 12:25:11.592622 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:11.592466 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:11.617858 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:11.617814 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" podStartSLOduration=1.6178018509999998 podStartE2EDuration="1.617801851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 12:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 12:25:11.615248433 +0000 UTC m=+2844.691103022" watchObservedRunningTime="2026-07-14 12:25:11.617801851 +0000 UTC m=+2844.693656439" Jul 14 12:25:12.529845 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:12.529816 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-lnt4q_91564ec5-d937-4fb6-a637-04f8a18975bd/volume-data-source-validator/0.log" Jul 14 12:25:13.718419 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:13.718387 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-246h5_ec672a19-4fb9-410f-b2c5-7b6ba92830e4/dns/0.log" Jul 14 12:25:13.744358 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:13.744321 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-246h5_ec672a19-4fb9-410f-b2c5-7b6ba92830e4/kube-rbac-proxy/0.log" Jul 14 12:25:13.966951 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:13.966917 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tfd9j_eed3a9e7-5412-4606-b01a-a852629061a5/dns-node-resolver/0.log" Jul 14 12:25:14.560781 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:14.560751 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c46ccf68d-5rwrx_86ef2bb0-98f3-4283-b43a-7e51484a79c7/registry/0.log" Jul 14 12:25:14.612011 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:14.611988 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dgl7z_1ff938fe-df60-4fb5-b558-e7832e86ff4b/node-ca/0.log" Jul 14 12:25:15.709055 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:15.709023 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qkhnv_0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497/istio-proxy/0.log" Jul 14 12:25:15.822941 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:15.822912 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2xhjw_010227ea-94b5-4e23-adce-6efe2c354add/discovery/0.log" Jul 14 12:25:15.992300 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:15.992276 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-tp2ll_aaf74151-7087-4688-b78f-713f36f0a271/payload-pre-processing/0.log" Jul 14 12:25:16.060199 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:16.060167 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7894875846-lcf2d_3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4/router/0.log" Jul 14 12:25:16.797730 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:16.797701 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5bdxl_96d809e2-9c08-4cb9-8a57-2c3aef340d37/serve-healthcheck-canary/0.log" Jul 14 12:25:17.606459 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:17.606432 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t6fcf/perf-node-gather-daemonset-rk89w" Jul 14 12:25:17.626556 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:17.626533 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9qsj_123b2e42-1ec7-459b-ae70-872058d5465e/kube-rbac-proxy/0.log" Jul 14 12:25:17.656497 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:17.656479 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9qsj_123b2e42-1ec7-459b-ae70-872058d5465e/exporter/0.log" Jul 14 12:25:17.682833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:17.682805 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9qsj_123b2e42-1ec7-459b-ae70-872058d5465e/extractor/0.log" Jul 14 12:25:20.803507 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:20.803475 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c944d8cbf-f4dct_4bc000c4-a69a-4777-90f5-36ec02e03702/manager/0.log" Jul 14 12:25:20.832842 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:20.832816 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c944d8cbf-f4dct_4bc000c4-a69a-4777-90f5-36ec02e03702/copy-manifests/0.log" Jul 14 12:25:20.877595 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:20.877564 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qkhnv_0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497/istio-proxy/0.log" Jul 14 12:25:21.002912 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:21.002883 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2xhjw_010227ea-94b5-4e23-adce-6efe2c354add/discovery/0.log" Jul 14 12:25:21.123833 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:21.123751 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-tp2ll_aaf74151-7087-4688-b78f-713f36f0a271/payload-pre-processing/0.log" Jul 14 12:25:21.168584 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:21.168561 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7894875846-lcf2d_3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4/router/0.log" Jul 14 12:25:22.343575 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:22.343548 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6ff9db667-nx6gp_9711ecf6-5274-45b6-965a-e622fb248319/manager/0.log" Jul 14 12:25:22.374138 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:22.374101 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-c8wvj_7ffbe918-4d30-449d-b0d7-1b8e63415230/openshift-lws-operator/0.log" Jul 14 12:25:22.481752 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:22.481723 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qkhnv_0dd2fa0d-6a7a-4c9c-9342-c2ede27a8497/istio-proxy/0.log" Jul 14 12:25:22.590519 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:22.590491 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2xhjw_010227ea-94b5-4e23-adce-6efe2c354add/discovery/0.log" Jul 14 12:25:22.709231 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:22.709147 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-tp2ll_aaf74151-7087-4688-b78f-713f36f0a271/payload-pre-processing/0.log" Jul 14 12:25:22.764067 ip-10-0-132-116 kubenswrapper[2562]: I0714 12:25:22.764038 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7894875846-lcf2d_3a0972d2-99aa-4b5a-9ba0-a3d64289f8f4/router/0.log"