Jul 24 14:56:20.940489 ip-10-0-139-100 systemd[1]: Starting Kubernetes Kubelet... Jul 24 14:56:21.299911 ip-10-0-139-100 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 14:56:21.299911 ip-10-0-139-100 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 14:56:21.299911 ip-10-0-139-100 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 14:56:21.299911 ip-10-0-139-100 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 14:56:21.299911 ip-10-0-139-100 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 14:56:21.300911 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.300795 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 14:56:21.305230 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305214 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 14:56:21.305230 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305229 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305233 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305237 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305240 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305244 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305246 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305249 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305252 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305254 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305257 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305260 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305263 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305265 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305268 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 14:56:21.305305 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.305270 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 14:56:21.307251 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307221 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 14:56:21.307500 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307482 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 14:56:21.307500 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307498 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 14:56:21.307500 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307502 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 14:56:21.307500 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307505 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307509 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307512 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307515 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307518 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307521 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307524 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307527 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307530 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307532 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307535 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307538 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307541 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307544 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307547 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307549 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307552 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307554 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307557 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307559 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 14:56:21.307650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307562 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307564 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307567 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307573 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307576 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307578 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307581 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307583 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307586 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307593 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307596 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307598 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307601 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307604 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307607 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307611 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307613 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307616 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307618 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 14:56:21.308164 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307621 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307624 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307626 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307631 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307636 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307641 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307644 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307647 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307650 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307653 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307656 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307660 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307663 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307665 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307668 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307671 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307675 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307681 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307684 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 14:56:21.308627 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307687 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307690 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307693 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307695 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307699 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307701 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307704 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307706 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307711 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307714 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307716 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307719 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307722 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307724 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307727 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307730 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307732 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307735 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307738 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307741 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 14:56:21.309083 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.307743 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308491 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308497 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308500 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308503 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308506 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308508 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308511 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308516 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308519 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308523 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308525 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308528 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308531 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308533 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308536 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308539 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308541 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308544 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308547 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 14:56:21.309650 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308550 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308553 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308555 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308558 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308561 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308563 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308566 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308568 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308571 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308573 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308575 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308578 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308580 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308583 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308589 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308592 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308595 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308598 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308601 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308603 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 14:56:21.310135 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308606 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308608 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308611 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308614 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308617 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308619 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308622 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308625 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308627 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308630 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308633 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308635 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308638 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308641 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308644 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308646 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308649 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308651 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308654 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 14:56:21.310637 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308656 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308658 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308661 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308664 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308666 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308668 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308671 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308673 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308676 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308678 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308681 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308684 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308688 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308691 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308694 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308697 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308701 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308703 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308706 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 14:56:21.311097 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308708 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308710 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308713 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308715 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308718 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308721 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308724 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308726 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308729 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308731 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308733 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308736 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308738 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308741 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308743 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308745 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308748 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308750 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308753 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308755 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 14:56:21.311561 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308759 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.308762 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309209 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309215 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309218 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309221 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309225 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309227 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309230 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309233 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309235 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309238 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309241 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309243 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309246 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309248 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309251 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309254 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309257 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309260 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 14:56:21.312030 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309262 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309265 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309267 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309270 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309272 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309274 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309277 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309279 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309282 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309284 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309286 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309289 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309291 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309294 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309297 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309299 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309303 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309305 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309308 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309311 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 14:56:21.312554 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309313 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309316 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309318 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309320 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309323 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309325 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309328 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309331 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309335 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309341 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309343 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309351 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309354 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309356 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309359 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309362 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309364 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309367 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309369 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 14:56:21.313033 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309372 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309374 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309376 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309379 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309381 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309384 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309386 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309388 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309392 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309395 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309398 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309400 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309403 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309406 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309408 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309410 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309414 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309417 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309420 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 14:56:21.313542 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309423 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309427 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309429 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309432 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309435 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309437 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309440 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309443 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309445 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309448 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309450 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309452 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309457 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309459 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309462 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309464 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309466 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309469 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309471 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309474 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 14:56:21.313998 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309478 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309481 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.309484 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309554 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309561 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309569 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309576 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309582 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309587 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309593 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309597 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309601 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309603 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309607 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309610 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309612 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309615 2557 flags.go:64] FLAG: --cgroup-root="" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309618 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309620 2557 flags.go:64] FLAG: --client-ca-file="" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309623 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309626 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309631 2557 flags.go:64] FLAG: --cluster-domain="" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309634 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 14:56:21.314493 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309637 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309641 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309644 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309648 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309651 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309655 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309658 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309661 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309664 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309667 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309670 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309672 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309678 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309683 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309688 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309691 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309694 2557 flags.go:64] FLAG: --enable-server="true" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309697 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309701 2557 flags.go:64] FLAG: --event-burst="100" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309704 2557 flags.go:64] FLAG: --event-qps="50" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309707 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309710 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309713 2557 flags.go:64] FLAG: --eviction-hard="" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309716 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 14:56:21.315089 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309719 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309722 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309725 2557 flags.go:64] FLAG: --eviction-soft="" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309728 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309730 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309733 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309736 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309739 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309741 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309746 2557 flags.go:64] FLAG: --feature-gates="" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309749 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309752 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309755 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309758 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309761 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309763 2557 flags.go:64] FLAG: --help="false" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309766 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309769 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309772 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309775 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309778 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309783 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309786 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 14:56:21.315665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309788 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309791 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309794 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309797 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309800 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309802 2557 flags.go:64] FLAG: --kube-reserved="" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309805 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309808 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309811 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309814 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309816 2557 flags.go:64] FLAG: --lock-file="" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309819 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309821 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309824 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309830 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309832 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309835 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309838 2557 flags.go:64] FLAG: --logging-format="text" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309841 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309845 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309848 2557 flags.go:64] FLAG: --manifest-url="" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309851 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309855 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309858 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309862 2557 flags.go:64] FLAG: --max-pods="110" Jul 24 14:56:21.316252 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309864 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309867 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309870 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309873 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309875 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309879 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309883 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309889 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309892 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309896 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309898 2557 flags.go:64] FLAG: --pod-cidr="" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309902 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309906 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309909 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309912 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309915 2557 flags.go:64] FLAG: --port="10250" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309917 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309920 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-051a2d80da6ff5275" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309923 2557 flags.go:64] FLAG: --qos-reserved="" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309926 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309929 2557 flags.go:64] FLAG: --register-node="true" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309931 2557 flags.go:64] FLAG: --register-with-taints="" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309935 2557 flags.go:64] FLAG: --registry-burst="10" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309937 2557 flags.go:64] FLAG: --registry-qps="5" Jul 24 14:56:21.316847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309940 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309944 2557 flags.go:64] FLAG: --reserved-memory="" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309948 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309951 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309954 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309957 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309960 2557 flags.go:64] FLAG: --runonce="false" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309962 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309965 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309968 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309971 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309973 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309976 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309980 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309984 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309987 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309990 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309993 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309996 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.309999 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310002 2557 flags.go:64] FLAG: --system-cgroups="" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310004 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310009 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310012 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310015 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 14:56:21.317442 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310019 2557 flags.go:64] FLAG: --tls-min-version="" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310022 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310024 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310027 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310030 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310032 2557 flags.go:64] FLAG: --v="2" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310036 2557 flags.go:64] FLAG: --version="false" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310040 2557 flags.go:64] FLAG: --vmodule="" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310046 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.310049 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310170 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310174 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310177 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310197 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310200 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310203 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310206 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310208 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310211 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310213 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310216 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310220 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 14:56:21.318039 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310222 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310225 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310228 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310231 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310233 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310236 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310238 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310240 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310243 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310245 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310248 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310251 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310254 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310256 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310259 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310261 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310264 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310266 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310273 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310277 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 14:56:21.318631 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310280 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310283 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310286 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310289 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310292 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310294 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310297 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310300 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310303 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310305 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310308 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310311 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310314 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310317 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310320 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310322 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310325 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310328 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310330 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310332 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 14:56:21.319126 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310335 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310338 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310340 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310343 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310345 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310348 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310350 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310353 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310355 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310359 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310363 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310366 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310368 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310371 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310373 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310376 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310378 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310381 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310384 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 14:56:21.319610 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310386 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310389 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310391 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310394 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310397 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310400 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310402 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310405 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310407 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310410 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310413 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310416 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310418 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310421 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310425 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310428 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310431 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310434 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310436 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 14:56:21.320063 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310439 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310442 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310444 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310446 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310450 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310452 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310455 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310457 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.310460 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.311194 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.318387 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.318402 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.319203 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.319211 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319265 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 14:56:21.320534 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319269 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319273 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319276 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319280 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319283 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319285 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319289 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319292 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319294 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319296 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319299 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319302 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319304 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319307 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319310 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319312 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319315 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319317 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319320 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319322 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 14:56:21.320902 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319325 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319328 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319330 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319334 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319337 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319339 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319341 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319345 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319349 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319351 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319354 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319357 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319359 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319362 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319365 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319368 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319370 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319373 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319376 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 14:56:21.321394 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319378 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319382 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319385 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319388 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319391 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319394 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319396 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319399 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319401 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319404 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319406 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319409 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319411 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319414 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319417 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319419 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319422 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319424 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319427 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 14:56:21.321845 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319430 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319432 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319435 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319437 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319440 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319443 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319445 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319447 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319450 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319452 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319455 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319457 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319460 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319462 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319465 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319467 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319470 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319472 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319475 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 14:56:21.322326 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319477 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319479 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319482 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319484 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319487 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319489 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319491 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319494 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319497 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319499 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319502 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319504 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319507 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319509 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319513 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319517 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319519 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319522 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319524 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 14:56:21.322793 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319526 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319529 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.319534 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319644 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319650 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319653 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319656 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319659 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319662 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319665 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319667 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319670 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319673 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319676 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 14:56:21.323257 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319680 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319683 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319686 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319688 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319690 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319693 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319695 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319698 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319700 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319702 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319705 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319707 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319710 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319712 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319715 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319717 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319720 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319722 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319724 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319727 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 14:56:21.323599 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319729 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319731 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319734 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319736 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319739 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319741 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319744 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319746 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319749 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319752 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319754 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319756 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319759 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319761 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319763 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319766 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319769 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319771 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319774 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319776 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 14:56:21.324113 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319779 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319781 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319788 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319791 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319794 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319797 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319800 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319803 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319806 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319808 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319811 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319813 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319816 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319818 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319820 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319823 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319826 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319828 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319830 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 14:56:21.324669 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319833 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319835 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319838 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319841 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319843 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319846 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319848 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319851 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319853 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319855 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319858 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319861 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319863 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319865 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319868 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319872 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319874 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319877 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319879 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319882 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 14:56:21.325120 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319884 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319887 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319889 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319892 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319894 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319896 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319899 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319901 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:21.319904 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.319909 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.320019 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.322339 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.323609 2557 server.go:1019] "Starting client certificate rotation" Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.323707 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 14:56:21.325618 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.323739 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 14:56:21.325997 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.325720 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 14:56:21.326935 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.326909 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 14:56:21.339736 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.339715 2557 log.go:25] "Validated CRI v1 runtime API" Jul 24 14:56:21.342075 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.342058 2557 log.go:25] "Validated CRI v1 image API" Jul 24 14:56:21.343990 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.343974 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 14:56:21.346258 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.346227 2557 fs.go:135] Filesystem UUIDs: map[13a84afc-d2c4-4631-b334-9614151db89e:/dev/nvme0n1p3 2c6d4db5-5c3a-4f8e-9bea-1745308f9408:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 24 14:56:21.346331 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.346252 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 14:56:21.352613 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.352485 2557 manager.go:216] Machine: {Timestamp:2026-07-24 14:56:21.351231049 +0000 UTC m=+0.298724627 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3186137 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20fce51ae294901763e2cb72d29861 SystemUUID:ec20fce5-1ae2-9490-1763-e2cb72d29861 BootID:eb009cfc-6bab-4da8-9e36-96d105686b30 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:83:7b:4c:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:83:7b:4c:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:64:5a:85:25:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 14:56:21.352613 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.352604 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 14:56:21.352785 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.352687 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 14:56:21.352840 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.352779 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 14:56:21.355562 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.355533 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 14:56:21.355701 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.355565 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 14:56:21.356146 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.356136 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 14:56:21.356202 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.356148 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 14:56:21.356202 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.356154 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 14:56:21.356801 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.356791 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 14:56:21.357938 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.357928 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 14:56:21.359679 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.359668 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 24 14:56:21.359785 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.359775 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 14:56:21.359834 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.359826 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 24 14:56:21.359863 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.359838 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 14:56:21.359863 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.359849 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 14:56:21.359863 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.359861 2557 kubelet.go:398] "Adding apiserver pod source" Jul 24 14:56:21.359953 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.359870 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 14:56:21.360258 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.360246 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 14:56:21.360305 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.360264 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 14:56:21.362266 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.362249 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 14:56:21.364441 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.364426 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 14:56:21.364528 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.364466 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 14:56:21.365965 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.365953 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 14:56:21.366046 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.365971 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 14:56:21.366046 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.365981 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 14:56:21.366046 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.365990 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 14:56:21.366046 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366002 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 14:56:21.366046 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366011 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 14:56:21.366046 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366022 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 14:56:21.366046 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366030 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 14:56:21.366046 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366041 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 14:56:21.366046 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366050 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 14:56:21.366353 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366062 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 14:56:21.366353 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366071 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 14:56:21.366732 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366720 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 14:56:21.366784 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.366735 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 14:56:21.370018 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.370002 2557 server.go:1268] "Started kubelet" Jul 24 14:56:21.370127 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.370094 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 14:56:21.373428 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.373047 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 14:56:21.373596 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.373578 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 14:56:21.373939 ip-10-0-139-100 systemd[1]: Started Kubernetes Kubelet. Jul 24 14:56:21.376731 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.376710 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 14:56:21.377575 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.377559 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 24 14:56:21.378225 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.378200 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 14:56:21.378298 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.378200 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 14:56:21.378298 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.378210 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 14:56:21.379055 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.378214 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-100.ec2.internal.18c5419fac61b5cd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-100.ec2.internal,UID:ip-10-0-139-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-100.ec2.internal,},FirstTimestamp:2026-07-24 14:56:21.369959885 +0000 UTC m=+0.317453461,LastTimestamp:2026-07-24 14:56:21.369959885 +0000 UTC m=+0.317453461,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-100.ec2.internal,}" Jul 24 14:56:21.381237 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.381218 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 14:56:21.385179 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.385158 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 14:56:21.385179 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.385157 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 14:56:21.385335 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.385256 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:21.385824 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.385803 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 14:56:21.385921 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.385805 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 14:56:21.385921 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.385845 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 14:56:21.386021 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.385940 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 24 14:56:21.386021 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.385949 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 24 14:56:21.386106 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.386091 2557 factory.go:55] Registering systemd factory Jul 24 14:56:21.386147 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.386107 2557 factory.go:223] Registration of the systemd container factory successfully Jul 24 14:56:21.387558 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.386356 2557 factory.go:153] Registering CRI-O factory Jul 24 14:56:21.387558 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.386373 2557 factory.go:223] Registration of the crio container factory successfully Jul 24 14:56:21.387558 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.386434 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 14:56:21.387558 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.386456 2557 factory.go:103] Registering Raw factory Jul 24 14:56:21.387558 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.386469 2557 manager.go:1196] Started watching for new ooms in manager Jul 24 14:56:21.387558 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.386863 2557 manager.go:349] Starting recovery of all containers Jul 24 14:56:21.388294 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.388273 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvb28" Jul 24 14:56:21.394933 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.394910 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvb28" Jul 24 14:56:21.396044 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.396012 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 14:56:21.396335 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.396151 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 14:56:21.398767 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.398750 2557 manager.go:354] Recovery completed Jul 24 14:56:21.405288 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.405272 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 14:56:21.407048 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.407032 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 14:56:21.407126 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.407062 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 14:56:21.407126 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.407073 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 14:56:21.407647 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.407631 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 14:56:21.407647 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.407645 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 14:56:21.407647 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.407646 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 14:56:21.407806 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.407659 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 24 14:56:21.408390 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.408375 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 14:56:21.408477 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.408407 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 14:56:21.408477 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.408422 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 14:56:21.409279 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.409268 2557 policy_none.go:49] "None policy: Start" Jul 24 14:56:21.409339 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.409283 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 14:56:21.409339 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.409293 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 14:56:21.410387 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.410377 2557 policy_none.go:47] "Start" Jul 24 14:56:21.441895 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.441746 2557 manager.go:337] "Starting Device Plugin manager" Jul 24 14:56:21.441991 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.441932 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 14:56:21.441991 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.441946 2557 server.go:87] "Starting device plugin registration server" Jul 24 14:56:21.442303 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.442290 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 14:56:21.442347 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.442307 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 14:56:21.442423 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.442405 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 14:56:21.442513 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.442500 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 14:56:21.442513 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.442512 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 14:56:21.443015 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.442998 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 14:56:21.443094 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.443049 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:21.543467 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.543441 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 14:56:21.544485 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.544470 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 14:56:21.544546 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.544503 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 14:56:21.544546 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.544513 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 14:56:21.544546 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.544541 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.553020 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.552964 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.553020 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.552989 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-100.ec2.internal\": node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:21.569768 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.569739 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:21.580011 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.579982 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 14:56:21.581420 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.581400 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 14:56:21.581516 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.581444 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 14:56:21.581516 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.581470 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 14:56:21.581516 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.581512 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 14:56:21.584377 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.584359 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 14:56:21.670072 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.670031 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:21.682263 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.682235 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"] Jul 24 14:56:21.682355 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.682313 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 14:56:21.683877 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.683859 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 14:56:21.683965 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.683894 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 14:56:21.683965 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.683908 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 14:56:21.685146 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.685131 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 14:56:21.685307 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.685293 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.685366 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.685322 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 14:56:21.685851 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.685827 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 14:56:21.685851 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.685852 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 14:56:21.685995 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.685865 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 14:56:21.685995 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.685893 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 14:56:21.685995 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.685910 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 14:56:21.685995 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.685921 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 14:56:21.687096 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.687080 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.687178 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.687112 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 14:56:21.687258 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.687228 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c64a1578ff1f9e01c1a10c531190df86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"c64a1578ff1f9e01c1a10c531190df86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.687312 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.687273 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c64a1578ff1f9e01c1a10c531190df86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"c64a1578ff1f9e01c1a10c531190df86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.687391 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.687328 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3ba7cef95532ae9bb2420cca3ac89de-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"a3ba7cef95532ae9bb2420cca3ac89de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.687762 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.687748 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory" Jul 24 14:56:21.687814 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.687774 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 14:56:21.687814 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.687789 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID" Jul 24 14:56:21.711796 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.711767 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.716110 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.716092 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.770588 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.770558 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:21.788116 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.788090 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3ba7cef95532ae9bb2420cca3ac89de-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"a3ba7cef95532ae9bb2420cca3ac89de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.788222 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.788125 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c64a1578ff1f9e01c1a10c531190df86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"c64a1578ff1f9e01c1a10c531190df86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.788222 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.788142 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c64a1578ff1f9e01c1a10c531190df86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"c64a1578ff1f9e01c1a10c531190df86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.788222 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.788203 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c64a1578ff1f9e01c1a10c531190df86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"c64a1578ff1f9e01c1a10c531190df86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.788404 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.788226 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3ba7cef95532ae9bb2420cca3ac89de-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"a3ba7cef95532ae9bb2420cca3ac89de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.788404 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:21.788243 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c64a1578ff1f9e01c1a10c531190df86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"c64a1578ff1f9e01c1a10c531190df86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 24 14:56:21.871126 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.871032 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:21.971594 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:21.971565 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:22.015158 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.015127 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 24 14:56:22.020440 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.020418 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 24 14:56:22.071981 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:22.071945 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:22.172561 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:22.172488 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:22.273105 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:22.273074 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:22.323545 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.323518 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 14:56:22.324008 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.323707 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 14:56:22.373999 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:22.373960 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:22.400836 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.400807 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 14:51:22 +0000 UTC" deadline="2028-03-15 19:30:47.340961895 +0000 UTC" Jul 24 14:56:22.400836 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.400832 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14404h34m24.940133254s" Jul 24 14:56:22.474134 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:22.474063 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:22.499273 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.499238 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 14:56:22.574567 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:22.574530 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found" Jul 24 14:56:22.591755 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.591736 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 14:56:22.609154 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:22.609120 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ba7cef95532ae9bb2420cca3ac89de.slice/crio-619c0f3b03c4f3dfdf133a8d312b273d433c737ae1481f8ec6612fcf3df1c5b1 WatchSource:0}: Error finding container 619c0f3b03c4f3dfdf133a8d312b273d433c737ae1481f8ec6612fcf3df1c5b1: Status 404 returned error can't find the container with id 619c0f3b03c4f3dfdf133a8d312b273d433c737ae1481f8ec6612fcf3df1c5b1 Jul 24 14:56:22.609441 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:22.609422 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64a1578ff1f9e01c1a10c531190df86.slice/crio-d9a02a43924425eb94e80e3b38b52ddcddc5a945d89d1a295a7aed28b417c7f9 WatchSource:0}: Error finding container d9a02a43924425eb94e80e3b38b52ddcddc5a945d89d1a295a7aed28b417c7f9: Status 404 returned error can't find the container with id d9a02a43924425eb94e80e3b38b52ddcddc5a945d89d1a295a7aed28b417c7f9 Jul 24 14:56:22.617430 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.617408 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 14:56:22.686198 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.686165 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" Jul 24 14:56:22.698679 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.698652 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 14:56:22.699480 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.699466 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" Jul 24 14:56:22.707431 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.707416 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 14:56:22.726924 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:22.726873 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 14:56:23.360935 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.360901 2557 apiserver.go:52] "Watching apiserver" Jul 24 14:56:23.369604 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.369580 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 14:56:23.370061 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.370037 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ghk9t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l","openshift-dns/node-resolver-t5h8r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal","openshift-multus/multus-additional-cni-plugins-jd6nn","openshift-multus/multus-qz268","openshift-multus/network-metrics-daemon-6gjqc","kube-system/konnectivity-agent-mgxft","kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal","openshift-cluster-node-tuning-operator/tuned-v7b8s","openshift-image-registry/node-ca-s2rlr","openshift-network-diagnostics/network-check-target-2hd8j","openshift-network-operator/iptables-alerter-vzvg5","openshift-ovn-kubernetes/ovnkube-node-vt4v8"] Jul 24 14:56:23.372832 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.372809 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.375320 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.375281 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.375519 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.375493 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wffbf\"" Jul 24 14:56:23.375618 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.375496 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 14:56:23.375893 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.375873 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 14:56:23.377677 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.377656 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 14:56:23.377773 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.377747 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 14:56:23.377998 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.377979 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ml8nh\"" Jul 24 14:56:23.377998 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.377993 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 14:56:23.379293 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.379274 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.381247 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.381221 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.384002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.383982 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.385297 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.385279 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 14:56:23.386056 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386037 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386424 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qz268" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386467 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386514 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d85jv\"" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386550 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386779 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386795 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386829 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386931 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386980 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 14:56:23.387002 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.386985 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 14:56:23.387588 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.387015 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 14:56:23.387588 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.387014 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 14:56:23.387588 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.387142 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 14:56:23.387588 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.387575 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 14:56:23.387781 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.387661 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-27c2d\"" Jul 24 14:56:23.387781 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.387720 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6qhj\"" Jul 24 14:56:23.389142 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.389121 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 14:56:23.389295 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.389278 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:23.391553 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.390860 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rt8j5\"" Jul 24 14:56:23.391756 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.391619 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 14:56:23.391812 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.391787 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ptph8\"" Jul 24 14:56:23.392369 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.392349 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 14:56:23.394142 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.394121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:23.394741 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.394397 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:23.395869 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.395843 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-system-cni-dir\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.395948 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.395874 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-sysctl-d\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.395948 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.395904 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.396039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.395955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/775e609b-1986-40e4-b87b-e01a2084f7a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.396039 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.395997 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f95e57-a279-4df4-a232-1d264a35c3fb-tmp-dir\") pod \"node-resolver-t5h8r\" (UID: \"a5f95e57-a279-4df4-a232-1d264a35c3fb\") " pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.396142 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d7dd653-a238-45b8-a1cb-53e871663fc6-ovnkube-config\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.396142 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396072 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whq2b\" (UniqueName: \"kubernetes.io/projected/3d7dd653-a238-45b8-a1cb-53e871663fc6-kube-api-access-whq2b\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.396142 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396110 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-run-multus-certs\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.396142 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396134 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-slash\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.396324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396157 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d7dd653-a238-45b8-a1cb-53e871663fc6-env-overrides\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.396324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396182 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/775e609b-1986-40e4-b87b-e01a2084f7a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.396324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396225 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-run-k8s-cni-cncf-io\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.396324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396250 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5f95e57-a279-4df4-a232-1d264a35c3fb-hosts-file\") pod \"node-resolver-t5h8r\" (UID: \"a5f95e57-a279-4df4-a232-1d264a35c3fb\") " pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.396324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396293 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdnvh\" (UniqueName: \"kubernetes.io/projected/775e609b-1986-40e4-b87b-e01a2084f7a5-kube-api-access-gdnvh\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.396567 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396343 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-node-log\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.396567 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396382 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-system-cni-dir\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.396567 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396409 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-os-release\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.396567 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396440 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-os-release\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.396567 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396466 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-var-lib-kubelet\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.396567 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396491 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-systemd\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.396567 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396514 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-socket-dir-parent\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.396567 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396557 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-etc-kubernetes\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396585 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-lib-modules\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396618 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjss\" (UniqueName: \"kubernetes.io/projected/1e68a653-4840-4e05-a102-6f514ee6c976-kube-api-access-hqjss\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396644 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edebd1c6-6669-4248-9283-b1477d194255-host-slash\") pod \"iptables-alerter-vzvg5\" (UID: \"edebd1c6-6669-4248-9283-b1477d194255\") " pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396662 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396675 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-conf-dir\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396701 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-modprobe-d\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396738 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-kubelet\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396760 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-var-lib-cni-bin\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396787 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-host\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e68a653-4840-4e05-a102-6f514ee6c976-etc-tuned\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396831 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9snpt\" (UniqueName: \"kubernetes.io/projected/a5f95e57-a279-4df4-a232-1d264a35c3fb-kube-api-access-9snpt\") pod \"node-resolver-t5h8r\" (UID: \"a5f95e57-a279-4df4-a232-1d264a35c3fb\") " pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396846 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-cni-netd\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396861 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-cni-binary-copy\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.396879 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396876 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87cf\" (UniqueName: \"kubernetes.io/projected/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-kube-api-access-c87cf\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396891 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77kh\" (UniqueName: \"kubernetes.io/projected/edebd1c6-6669-4248-9283-b1477d194255-kube-api-access-f77kh\") pod \"iptables-alerter-vzvg5\" (UID: \"edebd1c6-6669-4248-9283-b1477d194255\") " pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396906 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-run-openvswitch\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396921 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-cnibin\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396941 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-run-netns\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396956 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-hostroot\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396970 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-sys\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.396985 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-cni-dir\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397040 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-cnibin\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397071 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-var-lib-cni-multus\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397104 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-daemon-config\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397130 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e68a653-4840-4e05-a102-6f514ee6c976-tmp\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397154 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-systemd-units\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-run-netns\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397200 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d7dd653-a238-45b8-a1cb-53e871663fc6-ovnkube-script-lib\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397231 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-sysconfig\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397287 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-etc-openvswitch\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.397525 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397311 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/775e609b-1986-40e4-b87b-e01a2084f7a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397338 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-sysctl-conf\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397362 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-var-lib-kubelet\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397413 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397445 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397472 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-run\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397509 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-run-systemd\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397532 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-run-ovn\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397547 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-log-socket\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397571 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-cni-bin\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397602 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-kubernetes\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397618 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/edebd1c6-6669-4248-9283-b1477d194255-iptables-alerter-script\") pod \"iptables-alerter-vzvg5\" (UID: \"edebd1c6-6669-4248-9283-b1477d194255\") " pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397641 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-var-lib-openvswitch\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.398196 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.397665 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d7dd653-a238-45b8-a1cb-53e871663fc6-ovn-node-metrics-cert\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.399038 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.399018 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:23.399120 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.399093 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:23.399198 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.399131 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 14:56:23.399469 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.399449 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 14:56:23.399608 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.399520 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 14:56:23.399851 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.399692 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4l2hg\"" Jul 24 14:56:23.401596 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.401577 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.403055 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.403032 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 14:56:23.403894 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.403818 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 14:56:23.404011 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.403993 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 14:56:23.404179 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.404160 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mns5z\"" Jul 24 14:56:23.404279 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.404195 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 14:56:23.405077 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.405058 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:23.405268 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.405129 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:23.430624 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.430605 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pvslh" Jul 24 14:56:23.440466 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.440447 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pvslh" Jul 24 14:56:23.486863 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.486840 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 14:56:23.497927 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.497895 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-etc-kubernetes\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.498068 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.497947 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25d523dd-9ec8-4ae7-84f7-22c587ab5fbd-agent-certs\") pod \"konnectivity-agent-mgxft\" (UID: \"25d523dd-9ec8-4ae7-84f7-22c587ab5fbd\") " pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:23.498068 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.497977 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58n2m\" (UniqueName: \"kubernetes.io/projected/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-kube-api-access-58n2m\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.498068 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498006 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-lib-modules\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.498068 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498018 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-etc-kubernetes\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.498068 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498038 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjss\" (UniqueName: \"kubernetes.io/projected/1e68a653-4840-4e05-a102-6f514ee6c976-kube-api-access-hqjss\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.498068 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498066 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edebd1c6-6669-4248-9283-b1477d194255-host-slash\") pod \"iptables-alerter-vzvg5\" (UID: \"edebd1c6-6669-4248-9283-b1477d194255\") " pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498095 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-conf-dir\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498121 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498148 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498151 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edebd1c6-6669-4248-9283-b1477d194255-host-slash\") pod \"iptables-alerter-vzvg5\" (UID: \"edebd1c6-6669-4248-9283-b1477d194255\") " pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498184 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-modprobe-d\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498154 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-lib-modules\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498208 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-kubelet\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498235 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-conf-dir\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498257 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-var-lib-cni-bin\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498279 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-kubelet\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498294 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-var-lib-cni-bin\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498331 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-host\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498336 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-modprobe-d\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498371 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e68a653-4840-4e05-a102-6f514ee6c976-etc-tuned\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.498395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498403 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9snpt\" (UniqueName: \"kubernetes.io/projected/a5f95e57-a279-4df4-a232-1d264a35c3fb-kube-api-access-9snpt\") pod \"node-resolver-t5h8r\" (UID: \"a5f95e57-a279-4df4-a232-1d264a35c3fb\") " pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498413 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-host\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498423 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-cni-netd\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498473 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-cni-netd\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498483 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-cni-binary-copy\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87cf\" (UniqueName: \"kubernetes.io/projected/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-kube-api-access-c87cf\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498600 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498648 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77kh\" (UniqueName: \"kubernetes.io/projected/edebd1c6-6669-4248-9283-b1477d194255-kube-api-access-f77kh\") pod \"iptables-alerter-vzvg5\" (UID: \"edebd1c6-6669-4248-9283-b1477d194255\") " pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498796 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-run-openvswitch\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498958 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-cnibin\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498990 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-run-openvswitch\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.498986 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-run-netns\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499063 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-cnibin\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.499165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499142 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-hostroot\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499172 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zhq\" (UniqueName: \"kubernetes.io/projected/79e24294-248d-45f7-98c3-19b260dfbe88-kube-api-access-24zhq\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499173 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-run-netns\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499200 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-socket-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499240 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-hostroot\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499273 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-sys\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499284 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-cni-binary-copy\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499300 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-cni-dir\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499330 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-sys\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499333 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-cnibin\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499380 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-var-lib-cni-multus\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499448 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-cnibin\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499449 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-daemon-config\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499492 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e68a653-4840-4e05-a102-6f514ee6c976-tmp\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499506 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-var-lib-cni-multus\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499540 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-cni-dir\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499520 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-systemd-units\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499614 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-systemd-units\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.499873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499620 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-run-netns\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499714 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d7dd653-a238-45b8-a1cb-53e871663fc6-ovnkube-script-lib\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499776 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-run-netns\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499769 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-dbus\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499829 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-sysconfig\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499857 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-etc-openvswitch\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499884 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/775e609b-1986-40e4-b87b-e01a2084f7a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499890 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-sysconfig\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499949 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-etc-openvswitch\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.499984 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500009 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-kubelet-config\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500038 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-sysctl-conf\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500078 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-var-lib-kubelet\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500141 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-daemon-config\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500362 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500381 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-sysctl-conf\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.500751 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500403 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-registration-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500437 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9d9k\" (UniqueName: \"kubernetes.io/projected/668e6b2b-8552-4dc8-a40d-3d7961a9b066-kube-api-access-c9d9k\") pod \"node-ca-s2rlr\" (UID: \"668e6b2b-8552-4dc8-a40d-3d7961a9b066\") " pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500478 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-var-lib-kubelet\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500506 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-run\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500540 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-run-systemd\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500551 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d7dd653-a238-45b8-a1cb-53e871663fc6-ovnkube-script-lib\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500437 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500581 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500664 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-run\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500681 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-run-ovn\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500728 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-log-socket\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500754 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-cni-bin\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500813 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-kubernetes\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500857 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-cni-bin\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500890 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/edebd1c6-6669-4248-9283-b1477d194255-iptables-alerter-script\") pod \"iptables-alerter-vzvg5\" (UID: \"edebd1c6-6669-4248-9283-b1477d194255\") " pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500910 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-log-socket\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500917 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-var-lib-openvswitch\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.501601 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500942 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d7dd653-a238-45b8-a1cb-53e871663fc6-ovn-node-metrics-cert\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500968 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-kubernetes\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500965 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-system-cni-dir\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500755 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-run-systemd\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501051 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-var-lib-openvswitch\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501203 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-system-cni-dir\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.500813 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-run-ovn\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501251 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/668e6b2b-8552-4dc8-a40d-3d7961a9b066-serviceca\") pod \"node-ca-s2rlr\" (UID: \"668e6b2b-8552-4dc8-a40d-3d7961a9b066\") " pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501291 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-sysctl-d\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501308 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501328 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/775e609b-1986-40e4-b87b-e01a2084f7a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501345 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f95e57-a279-4df4-a232-1d264a35c3fb-tmp-dir\") pod \"node-resolver-t5h8r\" (UID: \"a5f95e57-a279-4df4-a232-1d264a35c3fb\") " pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501360 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d7dd653-a238-45b8-a1cb-53e871663fc6-ovnkube-config\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501376 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whq2b\" (UniqueName: \"kubernetes.io/projected/3d7dd653-a238-45b8-a1cb-53e871663fc6-kube-api-access-whq2b\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-run-multus-certs\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501410 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-slash\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501426 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d7dd653-a238-45b8-a1cb-53e871663fc6-env-overrides\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.502471 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/775e609b-1986-40e4-b87b-e01a2084f7a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501456 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-run-k8s-cni-cncf-io\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/668e6b2b-8552-4dc8-a40d-3d7961a9b066-host\") pod \"node-ca-s2rlr\" (UID: \"668e6b2b-8552-4dc8-a40d-3d7961a9b066\") " pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501487 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5f95e57-a279-4df4-a232-1d264a35c3fb-hosts-file\") pod \"node-resolver-t5h8r\" (UID: \"a5f95e57-a279-4df4-a232-1d264a35c3fb\") " pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501490 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/775e609b-1986-40e4-b87b-e01a2084f7a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501522 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdnvh\" (UniqueName: \"kubernetes.io/projected/775e609b-1986-40e4-b87b-e01a2084f7a5-kube-api-access-gdnvh\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501552 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25d523dd-9ec8-4ae7-84f7-22c587ab5fbd-konnectivity-ca\") pod \"konnectivity-agent-mgxft\" (UID: \"25d523dd-9ec8-4ae7-84f7-22c587ab5fbd\") " pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501577 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-device-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501601 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-sys-fs\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.501903 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/edebd1c6-6669-4248-9283-b1477d194255-iptables-alerter-script\") pod \"iptables-alerter-vzvg5\" (UID: \"edebd1c6-6669-4248-9283-b1477d194255\") " pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.502038 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-node-log\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.502093 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-system-cni-dir\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.502127 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-os-release\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.502199 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-os-release\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.502238 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-var-lib-kubelet\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.502352 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-systemd\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.502443 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-socket-dir-parent\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.503363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.502578 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/775e609b-1986-40e4-b87b-e01a2084f7a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.502674 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-sysctl-d\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503037 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f95e57-a279-4df4-a232-1d264a35c3fb-tmp-dir\") pod \"node-resolver-t5h8r\" (UID: \"a5f95e57-a279-4df4-a232-1d264a35c3fb\") " pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503175 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/775e609b-1986-40e4-b87b-e01a2084f7a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503340 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e68a653-4840-4e05-a102-6f514ee6c976-etc-tuned\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503368 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503449 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-system-cni-dir\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503524 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/775e609b-1986-40e4-b87b-e01a2084f7a5-os-release\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503629 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d7dd653-a238-45b8-a1cb-53e871663fc6-ovnkube-config\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503623 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-var-lib-kubelet\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503697 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-run-k8s-cni-cncf-io\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503711 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5f95e57-a279-4df4-a232-1d264a35c3fb-hosts-file\") pod \"node-resolver-t5h8r\" (UID: \"a5f95e57-a279-4df4-a232-1d264a35c3fb\") " pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503797 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-multus-socket-dir-parent\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503866 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-os-release\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503900 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-host-slash\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503926 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d7dd653-a238-45b8-a1cb-53e871663fc6-node-log\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.503982 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-host-run-multus-certs\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.504064 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d7dd653-a238-45b8-a1cb-53e871663fc6-env-overrides\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.504192 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.504108 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e68a653-4840-4e05-a102-6f514ee6c976-etc-systemd\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.505441 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.505420 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e68a653-4840-4e05-a102-6f514ee6c976-tmp\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.506441 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.506417 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d7dd653-a238-45b8-a1cb-53e871663fc6-ovn-node-metrics-cert\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.510784 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.510764 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9snpt\" (UniqueName: \"kubernetes.io/projected/a5f95e57-a279-4df4-a232-1d264a35c3fb-kube-api-access-9snpt\") pod \"node-resolver-t5h8r\" (UID: \"a5f95e57-a279-4df4-a232-1d264a35c3fb\") " pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.511517 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.511471 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77kh\" (UniqueName: \"kubernetes.io/projected/edebd1c6-6669-4248-9283-b1477d194255-kube-api-access-f77kh\") pod \"iptables-alerter-vzvg5\" (UID: \"edebd1c6-6669-4248-9283-b1477d194255\") " pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.511829 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.511802 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87cf\" (UniqueName: \"kubernetes.io/projected/24f1ce8d-b468-4bf4-b4e7-4be92062a0f1-kube-api-access-c87cf\") pod \"multus-qz268\" (UID: \"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1\") " pod="openshift-multus/multus-qz268" Jul 24 14:56:23.512155 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.512066 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjss\" (UniqueName: \"kubernetes.io/projected/1e68a653-4840-4e05-a102-6f514ee6c976-kube-api-access-hqjss\") pod \"tuned-v7b8s\" (UID: \"1e68a653-4840-4e05-a102-6f514ee6c976\") " pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.513919 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.513899 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whq2b\" (UniqueName: \"kubernetes.io/projected/3d7dd653-a238-45b8-a1cb-53e871663fc6-kube-api-access-whq2b\") pod \"ovnkube-node-vt4v8\" (UID: \"3d7dd653-a238-45b8-a1cb-53e871663fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.514125 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.514104 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdnvh\" (UniqueName: \"kubernetes.io/projected/775e609b-1986-40e4-b87b-e01a2084f7a5-kube-api-access-gdnvh\") pod \"multus-additional-cni-plugins-jd6nn\" (UID: \"775e609b-1986-40e4-b87b-e01a2084f7a5\") " pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.532381 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.532354 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 14:56:23.587970 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.587917 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"a3ba7cef95532ae9bb2420cca3ac89de","Type":"ContainerStarted","Data":"619c0f3b03c4f3dfdf133a8d312b273d433c737ae1481f8ec6612fcf3df1c5b1"} Jul 24 14:56:23.589025 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.588983 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"c64a1578ff1f9e01c1a10c531190df86","Type":"ContainerStarted","Data":"d9a02a43924425eb94e80e3b38b52ddcddc5a945d89d1a295a7aed28b417c7f9"} Jul 24 14:56:23.603665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603635 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/668e6b2b-8552-4dc8-a40d-3d7961a9b066-host\") pod \"node-ca-s2rlr\" (UID: \"668e6b2b-8552-4dc8-a40d-3d7961a9b066\") " pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.603665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603668 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25d523dd-9ec8-4ae7-84f7-22c587ab5fbd-konnectivity-ca\") pod \"konnectivity-agent-mgxft\" (UID: \"25d523dd-9ec8-4ae7-84f7-22c587ab5fbd\") " pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:23.603878 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603690 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-device-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.603878 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603711 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-sys-fs\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.603878 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603731 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25d523dd-9ec8-4ae7-84f7-22c587ab5fbd-agent-certs\") pod \"konnectivity-agent-mgxft\" (UID: \"25d523dd-9ec8-4ae7-84f7-22c587ab5fbd\") " pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:23.603878 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603732 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/668e6b2b-8552-4dc8-a40d-3d7961a9b066-host\") pod \"node-ca-s2rlr\" (UID: \"668e6b2b-8552-4dc8-a40d-3d7961a9b066\") " pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.603878 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603748 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58n2m\" (UniqueName: \"kubernetes.io/projected/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-kube-api-access-58n2m\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.603878 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603784 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.603878 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603834 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-sys-fs\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.603878 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603863 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-device-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603890 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603913 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603951 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.603985 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604019 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zhq\" (UniqueName: \"kubernetes.io/projected/79e24294-248d-45f7-98c3-19b260dfbe88-kube-api-access-24zhq\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604047 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-socket-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.604056 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604081 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-dbus\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604123 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.604141 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret podName:bd1f81ff-a1c3-4714-bbec-423bdf6e06b9 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:24.104099251 +0000 UTC m=+3.051592825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret") pod "global-pull-secret-syncer-ghk9t" (UID: "bd1f81ff-a1c3-4714-bbec-423bdf6e06b9") : object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604150 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-dbus\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604162 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604191 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-kubelet-config\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604245 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-registration-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.604252 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604298 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-socket-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.604373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604330 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-registration-dir\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.605254 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.604338 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs podName:79e24294-248d-45f7-98c3-19b260dfbe88 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:24.10431977 +0000 UTC m=+3.051813355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs") pod "network-metrics-daemon-6gjqc" (UID: "79e24294-248d-45f7-98c3-19b260dfbe88") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:23.605254 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604274 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9d9k\" (UniqueName: \"kubernetes.io/projected/668e6b2b-8552-4dc8-a40d-3d7961a9b066-kube-api-access-c9d9k\") pod \"node-ca-s2rlr\" (UID: \"668e6b2b-8552-4dc8-a40d-3d7961a9b066\") " pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.605254 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604370 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-kubelet-config\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:23.605254 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/668e6b2b-8552-4dc8-a40d-3d7961a9b066-serviceca\") pod \"node-ca-s2rlr\" (UID: \"668e6b2b-8552-4dc8-a40d-3d7961a9b066\") " pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.605254 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604814 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25d523dd-9ec8-4ae7-84f7-22c587ab5fbd-konnectivity-ca\") pod \"konnectivity-agent-mgxft\" (UID: \"25d523dd-9ec8-4ae7-84f7-22c587ab5fbd\") " pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:23.605254 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.604846 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/668e6b2b-8552-4dc8-a40d-3d7961a9b066-serviceca\") pod \"node-ca-s2rlr\" (UID: \"668e6b2b-8552-4dc8-a40d-3d7961a9b066\") " pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.606306 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.606287 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25d523dd-9ec8-4ae7-84f7-22c587ab5fbd-agent-certs\") pod \"konnectivity-agent-mgxft\" (UID: \"25d523dd-9ec8-4ae7-84f7-22c587ab5fbd\") " pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:23.614990 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.614935 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 14:56:23.614990 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.614958 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 14:56:23.614990 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.614970 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8wjxh for pod openshift-network-diagnostics/network-check-target-2hd8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:23.615225 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:23.615027 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh podName:a1e6028f-648d-4a7b-9ce4-b817d1bff47b nodeName:}" failed. No retries permitted until 2026-07-24 14:56:24.115011058 +0000 UTC m=+3.062504623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8wjxh" (UniqueName: "kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh") pod "network-check-target-2hd8j" (UID: "a1e6028f-648d-4a7b-9ce4-b817d1bff47b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:23.617707 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.617684 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58n2m\" (UniqueName: \"kubernetes.io/projected/7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3-kube-api-access-58n2m\") pod \"aws-ebs-csi-driver-node-n5k2l\" (UID: \"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.618741 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.618715 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9d9k\" (UniqueName: \"kubernetes.io/projected/668e6b2b-8552-4dc8-a40d-3d7961a9b066-kube-api-access-c9d9k\") pod \"node-ca-s2rlr\" (UID: \"668e6b2b-8552-4dc8-a40d-3d7961a9b066\") " pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.619245 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.619227 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zhq\" (UniqueName: \"kubernetes.io/projected/79e24294-248d-45f7-98c3-19b260dfbe88-kube-api-access-24zhq\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:23.685281 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.685243 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" Jul 24 14:56:23.697829 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.697803 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vzvg5" Jul 24 14:56:23.705955 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.705926 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t5h8r" Jul 24 14:56:23.711897 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.711877 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:23.719775 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.719744 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" Jul 24 14:56:23.726752 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.726731 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qz268" Jul 24 14:56:23.734748 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.734725 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:23.742585 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.742566 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" Jul 24 14:56:23.748399 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:23.748377 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s2rlr" Jul 24 14:56:23.934224 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:23.934171 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f95e57_a279_4df4_a232_1d264a35c3fb.slice/crio-c5445ddf949ea756939a305425e0970896d64883052fcf171d0684711dedc407 WatchSource:0}: Error finding container c5445ddf949ea756939a305425e0970896d64883052fcf171d0684711dedc407: Status 404 returned error can't find the container with id c5445ddf949ea756939a305425e0970896d64883052fcf171d0684711dedc407 Jul 24 14:56:23.934842 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:23.934818 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f1ce8d_b468_4bf4_b4e7_4be92062a0f1.slice/crio-02bbe1f38478f332eac6707e83d8242c4493434ba4a845e820898c9d2ab71d18 WatchSource:0}: Error finding container 02bbe1f38478f332eac6707e83d8242c4493434ba4a845e820898c9d2ab71d18: Status 404 returned error can't find the container with id 02bbe1f38478f332eac6707e83d8242c4493434ba4a845e820898c9d2ab71d18 Jul 24 14:56:23.936060 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:23.936024 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d523dd_9ec8_4ae7_84f7_22c587ab5fbd.slice/crio-2193255b68756b2e285d6771a8c11fb41b527526984f4a445a234ac895453ccf WatchSource:0}: Error finding container 2193255b68756b2e285d6771a8c11fb41b527526984f4a445a234ac895453ccf: Status 404 returned error can't find the container with id 2193255b68756b2e285d6771a8c11fb41b527526984f4a445a234ac895453ccf Jul 24 14:56:23.936977 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:23.936595 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e68a653_4840_4e05_a102_6f514ee6c976.slice/crio-8f61746f71b2ddeab2394d31ca6b04686f3f8ccd51efe709d7db1a205d107b7e WatchSource:0}: Error finding container 8f61746f71b2ddeab2394d31ca6b04686f3f8ccd51efe709d7db1a205d107b7e: Status 404 returned error can't find the container with id 8f61746f71b2ddeab2394d31ca6b04686f3f8ccd51efe709d7db1a205d107b7e Jul 24 14:56:23.939805 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:23.939782 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668e6b2b_8552_4dc8_a40d_3d7961a9b066.slice/crio-bb046de807f76e9df49fbf5a8021152fbdec99776d2ccdd9d6b01c2c8051fe99 WatchSource:0}: Error finding container bb046de807f76e9df49fbf5a8021152fbdec99776d2ccdd9d6b01c2c8051fe99: Status 404 returned error can't find the container with id bb046de807f76e9df49fbf5a8021152fbdec99776d2ccdd9d6b01c2c8051fe99 Jul 24 14:56:23.941693 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:23.941668 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedebd1c6_6669_4248_9283_b1477d194255.slice/crio-2efa63ccc87275cf9b82b55e3763affafdb4eefafdf67c95ecf2d32a874f13c3 WatchSource:0}: Error finding container 2efa63ccc87275cf9b82b55e3763affafdb4eefafdf67c95ecf2d32a874f13c3: Status 404 returned error can't find the container with id 2efa63ccc87275cf9b82b55e3763affafdb4eefafdf67c95ecf2d32a874f13c3 Jul 24 14:56:23.942506 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:23.942470 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b74e50b_a842_4cc2_9022_9ebdb3dc6bf3.slice/crio-e0cd20a6bc35661a958dbbc2a37e2cca14f83e8edb99d532c20a035db8c10358 WatchSource:0}: Error finding container e0cd20a6bc35661a958dbbc2a37e2cca14f83e8edb99d532c20a035db8c10358: Status 404 returned error can't find the container with id e0cd20a6bc35661a958dbbc2a37e2cca14f83e8edb99d532c20a035db8c10358 Jul 24 14:56:24.108395 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.108360 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:24.108627 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.108434 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:24.108627 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:24.108552 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:24.108627 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:24.108583 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:24.108775 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:24.108636 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs podName:79e24294-248d-45f7-98c3-19b260dfbe88 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:25.108613581 +0000 UTC m=+4.056107164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs") pod "network-metrics-daemon-6gjqc" (UID: "79e24294-248d-45f7-98c3-19b260dfbe88") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:24.108775 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:24.108658 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret podName:bd1f81ff-a1c3-4714-bbec-423bdf6e06b9 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:25.108647772 +0000 UTC m=+4.056141346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret") pod "global-pull-secret-syncer-ghk9t" (UID: "bd1f81ff-a1c3-4714-bbec-423bdf6e06b9") : object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:24.208950 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.208923 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:24.209179 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:24.209111 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 14:56:24.209179 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:24.209135 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 14:56:24.209179 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:24.209148 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8wjxh for pod openshift-network-diagnostics/network-check-target-2hd8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:24.209299 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:24.209211 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh podName:a1e6028f-648d-4a7b-9ce4-b817d1bff47b nodeName:}" failed. No retries permitted until 2026-07-24 14:56:25.209192136 +0000 UTC m=+4.156685716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wjxh" (UniqueName: "kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh") pod "network-check-target-2hd8j" (UID: "a1e6028f-648d-4a7b-9ce4-b817d1bff47b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:24.444040 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.443686 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 14:51:24 +0000 UTC" deadline="2028-03-30 18:37:37.610412966 +0000 UTC" Jul 24 14:56:24.444040 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.443982 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14763h41m13.166436986s" Jul 24 14:56:24.582678 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.582638 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:24.582884 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:24.582779 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:24.613899 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.613767 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" event={"ID":"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3","Type":"ContainerStarted","Data":"e0cd20a6bc35661a958dbbc2a37e2cca14f83e8edb99d532c20a035db8c10358"} Jul 24 14:56:24.617808 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.617718 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" event={"ID":"775e609b-1986-40e4-b87b-e01a2084f7a5","Type":"ContainerStarted","Data":"c5560748610b904b79a4069bef1456e13211a81602e9cf7f34e4f48b8fff0827"} Jul 24 14:56:24.626494 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.626453 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s2rlr" event={"ID":"668e6b2b-8552-4dc8-a40d-3d7961a9b066","Type":"ContainerStarted","Data":"bb046de807f76e9df49fbf5a8021152fbdec99776d2ccdd9d6b01c2c8051fe99"} Jul 24 14:56:24.637139 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.636161 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" event={"ID":"1e68a653-4840-4e05-a102-6f514ee6c976","Type":"ContainerStarted","Data":"8f61746f71b2ddeab2394d31ca6b04686f3f8ccd51efe709d7db1a205d107b7e"} Jul 24 14:56:24.644036 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.644003 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mgxft" event={"ID":"25d523dd-9ec8-4ae7-84f7-22c587ab5fbd","Type":"ContainerStarted","Data":"2193255b68756b2e285d6771a8c11fb41b527526984f4a445a234ac895453ccf"} Jul 24 14:56:24.650874 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.650842 2557 generic.go:356] "Generic (PLEG): container finished" podID="c64a1578ff1f9e01c1a10c531190df86" containerID="16986e897c6674b8d574e26bbd7dab82e421375ddfc011ef46ef98ea6dd47321" exitCode=0 Jul 24 14:56:24.651003 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.650929 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"c64a1578ff1f9e01c1a10c531190df86","Type":"ContainerDied","Data":"16986e897c6674b8d574e26bbd7dab82e421375ddfc011ef46ef98ea6dd47321"} Jul 24 14:56:24.657462 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.657339 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" event={"ID":"3d7dd653-a238-45b8-a1cb-53e871663fc6","Type":"ContainerStarted","Data":"0d7ee7d9c66af6fcc8c65319384a241572b8f8a75a799dba601afb6eaf51550d"} Jul 24 14:56:24.661409 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.661381 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vzvg5" event={"ID":"edebd1c6-6669-4248-9283-b1477d194255","Type":"ContainerStarted","Data":"2efa63ccc87275cf9b82b55e3763affafdb4eefafdf67c95ecf2d32a874f13c3"} Jul 24 14:56:24.671049 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.671019 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qz268" event={"ID":"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1","Type":"ContainerStarted","Data":"02bbe1f38478f332eac6707e83d8242c4493434ba4a845e820898c9d2ab71d18"} Jul 24 14:56:24.674419 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:24.674374 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t5h8r" event={"ID":"a5f95e57-a279-4df4-a232-1d264a35c3fb","Type":"ContainerStarted","Data":"c5445ddf949ea756939a305425e0970896d64883052fcf171d0684711dedc407"} Jul 24 14:56:25.117965 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:25.117863 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:25.117965 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:25.117951 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:25.118243 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.118101 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:25.118243 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.118170 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs podName:79e24294-248d-45f7-98c3-19b260dfbe88 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:27.118150634 +0000 UTC m=+6.065644208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs") pod "network-metrics-daemon-6gjqc" (UID: "79e24294-248d-45f7-98c3-19b260dfbe88") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:25.118624 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.118602 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:25.118714 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.118667 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret podName:bd1f81ff-a1c3-4714-bbec-423bdf6e06b9 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:27.118648696 +0000 UTC m=+6.066142278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret") pod "global-pull-secret-syncer-ghk9t" (UID: "bd1f81ff-a1c3-4714-bbec-423bdf6e06b9") : object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:25.219037 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:25.218997 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:25.219345 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.219210 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 14:56:25.219345 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.219255 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 14:56:25.219345 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.219271 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8wjxh for pod openshift-network-diagnostics/network-check-target-2hd8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:25.219345 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.219333 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh podName:a1e6028f-648d-4a7b-9ce4-b817d1bff47b nodeName:}" failed. No retries permitted until 2026-07-24 14:56:27.219314648 +0000 UTC m=+6.166808213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wjxh" (UniqueName: "kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh") pod "network-check-target-2hd8j" (UID: "a1e6028f-648d-4a7b-9ce4-b817d1bff47b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:25.445161 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:25.445069 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 14:51:24 +0000 UTC" deadline="2028-04-17 03:14:29.779941392 +0000 UTC" Jul 24 14:56:25.445161 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:25.445108 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15180h18m4.334838202s" Jul 24 14:56:25.585121 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:25.584438 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:25.585121 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.584591 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:25.585121 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:25.584675 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:25.585121 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:25.584756 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:25.685645 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:25.685605 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"c64a1578ff1f9e01c1a10c531190df86","Type":"ContainerStarted","Data":"98b27b8c0d506a350d16b0fb5c58b77ac549022bb29b4152af4b96b413d874cb"} Jul 24 14:56:26.582513 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:26.582473 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:26.583013 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:26.582625 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:27.135688 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:27.135644 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:27.135887 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:27.135728 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:27.135887 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.135854 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:27.136048 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.135934 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs podName:79e24294-248d-45f7-98c3-19b260dfbe88 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:31.1359144 +0000 UTC m=+10.083407964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs") pod "network-metrics-daemon-6gjqc" (UID: "79e24294-248d-45f7-98c3-19b260dfbe88") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:27.136429 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.136409 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:27.136540 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.136466 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret podName:bd1f81ff-a1c3-4714-bbec-423bdf6e06b9 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:31.13645084 +0000 UTC m=+10.083944420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret") pod "global-pull-secret-syncer-ghk9t" (UID: "bd1f81ff-a1c3-4714-bbec-423bdf6e06b9") : object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:27.236610 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:27.236548 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:27.236804 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.236722 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 14:56:27.236804 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.236742 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 14:56:27.236804 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.236755 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8wjxh for pod openshift-network-diagnostics/network-check-target-2hd8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:27.236991 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.236821 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh podName:a1e6028f-648d-4a7b-9ce4-b817d1bff47b nodeName:}" failed. No retries permitted until 2026-07-24 14:56:31.236801046 +0000 UTC m=+10.184294626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wjxh" (UniqueName: "kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh") pod "network-check-target-2hd8j" (UID: "a1e6028f-648d-4a7b-9ce4-b817d1bff47b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:27.583463 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:27.583369 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:27.583463 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:27.583402 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:27.584154 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.583520 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:27.584154 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:27.583629 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:28.582472 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:28.582436 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:28.582622 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:28.582586 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:29.585067 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:29.585019 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:29.585590 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:29.585159 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:29.585590 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:29.585574 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:29.585711 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:29.585687 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:30.581871 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:30.581838 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:30.582064 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:30.581960 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:31.174218 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:31.174170 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:31.174786 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:31.174276 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:31.174786 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.174372 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:31.174786 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.174447 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret podName:bd1f81ff-a1c3-4714-bbec-423bdf6e06b9 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:39.174429726 +0000 UTC m=+18.121923292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret") pod "global-pull-secret-syncer-ghk9t" (UID: "bd1f81ff-a1c3-4714-bbec-423bdf6e06b9") : object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:31.174786 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.174379 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:31.174786 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.174540 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs podName:79e24294-248d-45f7-98c3-19b260dfbe88 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:39.174524921 +0000 UTC m=+18.122018506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs") pod "network-metrics-daemon-6gjqc" (UID: "79e24294-248d-45f7-98c3-19b260dfbe88") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:31.275624 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:31.275578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:31.275829 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.275791 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 14:56:31.275829 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.275821 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 14:56:31.275938 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.275834 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8wjxh for pod openshift-network-diagnostics/network-check-target-2hd8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:31.275938 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.275898 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh podName:a1e6028f-648d-4a7b-9ce4-b817d1bff47b nodeName:}" failed. No retries permitted until 2026-07-24 14:56:39.275877858 +0000 UTC m=+18.223371448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wjxh" (UniqueName: "kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh") pod "network-check-target-2hd8j" (UID: "a1e6028f-648d-4a7b-9ce4-b817d1bff47b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:31.583496 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:31.583385 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:31.583673 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.583566 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:31.585378 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:31.585347 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:31.585516 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:31.585459 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:32.582583 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:32.582540 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:32.583003 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:32.582656 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:33.583356 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:33.583323 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:33.583819 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:33.583471 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:33.583819 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:33.583810 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:33.583937 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:33.583916 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:34.582321 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:34.582292 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:34.582498 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:34.582418 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:35.582475 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:35.582435 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:35.582884 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:35.582489 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:35.582884 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:35.582591 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:35.582884 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:35.582715 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:36.582412 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:36.582377 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:36.582562 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:36.582503 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:37.582680 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:37.582648 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:37.583125 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:37.582657 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:37.583125 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:37.582782 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:37.583125 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:37.582846 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:38.581953 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:38.581919 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:38.582205 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:38.582033 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:39.239738 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:39.239695 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:39.240163 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:39.239754 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:39.240163 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.239856 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:39.240163 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.239931 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret podName:bd1f81ff-a1c3-4714-bbec-423bdf6e06b9 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:55.239916018 +0000 UTC m=+34.187409582 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret") pod "global-pull-secret-syncer-ghk9t" (UID: "bd1f81ff-a1c3-4714-bbec-423bdf6e06b9") : object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:39.240163 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.239857 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:39.240163 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.239994 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs podName:79e24294-248d-45f7-98c3-19b260dfbe88 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:55.239981687 +0000 UTC m=+34.187475251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs") pod "network-metrics-daemon-6gjqc" (UID: "79e24294-248d-45f7-98c3-19b260dfbe88") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:39.340678 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:39.340645 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:39.340834 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.340781 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 14:56:39.340834 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.340797 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 14:56:39.340834 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.340807 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8wjxh for pod openshift-network-diagnostics/network-check-target-2hd8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:39.340930 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.340860 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh podName:a1e6028f-648d-4a7b-9ce4-b817d1bff47b nodeName:}" failed. No retries permitted until 2026-07-24 14:56:55.340844839 +0000 UTC m=+34.288338404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wjxh" (UniqueName: "kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh") pod "network-check-target-2hd8j" (UID: "a1e6028f-648d-4a7b-9ce4-b817d1bff47b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:39.582877 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:39.582798 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:39.583022 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.582921 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:39.583101 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:39.582799 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:39.583231 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:39.583179 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:40.582377 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:40.582338 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:40.582853 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:40.582471 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:41.584583 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:41.584545 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:41.585034 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:41.584698 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:41.585092 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:41.585055 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:41.585165 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:41.585138 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:42.581947 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:42.581910 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:42.582131 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:42.582018 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:43.581957 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:43.581921 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:43.582507 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:43.581969 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:43.582507 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:43.582070 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:43.582507 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:43.582172 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:44.581920 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.581741 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:44.582053 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:44.581991 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:44.730750 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.730711 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" event={"ID":"3d7dd653-a238-45b8-a1cb-53e871663fc6","Type":"ContainerStarted","Data":"ef92ce96e11d3c5b4519775419d836181a0626ffcd86a982c016d47547e14cd4"} Jul 24 14:56:44.730967 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.730790 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" event={"ID":"3d7dd653-a238-45b8-a1cb-53e871663fc6","Type":"ContainerStarted","Data":"dc663611e27be2a6c1f23879c18a138f4cf072563a49aa30dc7475a24effa357"} Jul 24 14:56:44.730967 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.730829 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" event={"ID":"3d7dd653-a238-45b8-a1cb-53e871663fc6","Type":"ContainerStarted","Data":"4a8d65575f72491c1fc8c71519dfdd8356d69e79b4ddcaef24142186404f4f22"} Jul 24 14:56:44.730967 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.730861 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" event={"ID":"3d7dd653-a238-45b8-a1cb-53e871663fc6","Type":"ContainerStarted","Data":"05e19c023badc06ea0572d81af139a1e1ecbbb395cb49fb63a6a0f64ccec3f90"} Jul 24 14:56:44.730967 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.730888 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" event={"ID":"3d7dd653-a238-45b8-a1cb-53e871663fc6","Type":"ContainerStarted","Data":"46ec8f5fc539ec2d8cf1471dfc71c3c6c7d6ba8bb2a482099352e10d5ef7c4e5"} Jul 24 14:56:44.730967 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.730921 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" event={"ID":"3d7dd653-a238-45b8-a1cb-53e871663fc6","Type":"ContainerStarted","Data":"cbeb510aecaabb3d6e2274d4bc8dec934bd3d0ec62e209d949bd6131b615eed2"} Jul 24 14:56:44.732322 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.732295 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qz268" event={"ID":"24f1ce8d-b468-4bf4-b4e7-4be92062a0f1","Type":"ContainerStarted","Data":"b564cef16da568b5a3ab925aadd02dc9110c8b214997040e246184811d76a497"} Jul 24 14:56:44.733785 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.733763 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t5h8r" event={"ID":"a5f95e57-a279-4df4-a232-1d264a35c3fb","Type":"ContainerStarted","Data":"07b7df851f3fc0389cac1b12f284b6facbfbd3e6ca582d50aeb2c74fe565ae33"} Jul 24 14:56:44.735224 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.735198 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" event={"ID":"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3","Type":"ContainerStarted","Data":"bbb9f79528a3aa0e6620ed59787e4f59b4302757cdb2f9b852eccdbc836b9dde"} Jul 24 14:56:44.736519 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.736497 2557 generic.go:356] "Generic (PLEG): container finished" podID="775e609b-1986-40e4-b87b-e01a2084f7a5" containerID="8e3c1b823ffe26ccaafa7252d3fc5e06cdda7838e015099e934a508189fa4da4" exitCode=0 Jul 24 14:56:44.736626 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.736553 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" event={"ID":"775e609b-1986-40e4-b87b-e01a2084f7a5","Type":"ContainerDied","Data":"8e3c1b823ffe26ccaafa7252d3fc5e06cdda7838e015099e934a508189fa4da4"} Jul 24 14:56:44.737964 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.737939 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s2rlr" event={"ID":"668e6b2b-8552-4dc8-a40d-3d7961a9b066","Type":"ContainerStarted","Data":"d61ba17df256c658c844fde5cf70fc189454d57ae608abf151b146054fc2d3de"} Jul 24 14:56:44.739382 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.739362 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" event={"ID":"1e68a653-4840-4e05-a102-6f514ee6c976","Type":"ContainerStarted","Data":"4c38024555b8118af2bcb8da5652ac6f60522c201ce179df5b1d4bbbc5467a81"} Jul 24 14:56:44.740829 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.740771 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mgxft" event={"ID":"25d523dd-9ec8-4ae7-84f7-22c587ab5fbd","Type":"ContainerStarted","Data":"b42e26d203dfa8fa3cc060f1bc72256138bcc39863ea5880b24d20dcf0a67bf5"} Jul 24 14:56:44.741886 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.741862 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"a3ba7cef95532ae9bb2420cca3ac89de","Type":"ContainerStarted","Data":"2eb334eaf0be5486bdd636e32cfc0afc31c66103c6119f985ef784a0856dd9d5"} Jul 24 14:56:44.751526 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.751491 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" podStartSLOduration=21.751478752 podStartE2EDuration="21.751478752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:25.707175927 +0000 UTC m=+4.654669518" watchObservedRunningTime="2026-07-24 14:56:44.751478752 +0000 UTC m=+23.698972334" Jul 24 14:56:44.751652 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.751631 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qz268" podStartSLOduration=2.731070233 podStartE2EDuration="22.751624922s" totalImagesPullingTime="20.020554689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:44.749999931 +0000 UTC m=+23.697493517" watchObservedRunningTime="2026-07-24 14:56:44.751624922 +0000 UTC m=+23.699118507" Jul 24 14:56:44.782410 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.782369 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" podStartSLOduration=21.78235602 podStartE2EDuration="21.78235602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:44.781995208 +0000 UTC m=+23.729488793" watchObservedRunningTime="2026-07-24 14:56:44.78235602 +0000 UTC m=+23.729849605" Jul 24 14:56:44.782517 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.782472 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s2rlr" podStartSLOduration=2.7708721450000002 podStartE2EDuration="22.782465716s" totalImagesPullingTime="20.011593571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:44.76725773 +0000 UTC m=+23.714751330" watchObservedRunningTime="2026-07-24 14:56:44.782465716 +0000 UTC m=+23.729959301" Jul 24 14:56:44.800825 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.800787 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v7b8s" podStartSLOduration=2.784277095 podStartE2EDuration="22.800775112s" totalImagesPullingTime="20.016498017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:44.800317685 +0000 UTC m=+23.747811271" watchObservedRunningTime="2026-07-24 14:56:44.800775112 +0000 UTC m=+23.748268698" Jul 24 14:56:44.856613 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.856552 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t5h8r" podStartSLOduration=3.132093154 podStartE2EDuration="22.85653915s" totalImagesPullingTime="19.724445996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:44.826681997 +0000 UTC m=+23.774175582" watchObservedRunningTime="2026-07-24 14:56:44.85653915 +0000 UTC m=+23.804032802" Jul 24 14:56:44.869609 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:44.869569 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mgxft" podStartSLOduration=11.354832763 podStartE2EDuration="22.869555506s" totalImagesPullingTime="11.514722743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:44.869262234 +0000 UTC m=+23.816755821" watchObservedRunningTime="2026-07-24 14:56:44.869555506 +0000 UTC m=+23.817049091" Jul 24 14:56:45.582096 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:45.582020 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:45.582501 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:45.582166 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:45.582501 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:45.582213 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:45.582501 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:45.582309 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:45.746337 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:45.746293 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vzvg5" event={"ID":"edebd1c6-6669-4248-9283-b1477d194255","Type":"ContainerStarted","Data":"a8f8459d9d62edb91e46a1a0d062fce7b56ca346f05a8bbf28152e24f795505c"} Jul 24 14:56:45.763773 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:45.763732 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vzvg5" podStartSLOduration=3.752443361 podStartE2EDuration="23.763717515s" totalImagesPullingTime="20.011274154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:45.763524234 +0000 UTC m=+24.711017820" watchObservedRunningTime="2026-07-24 14:56:45.763717515 +0000 UTC m=+24.711211126" Jul 24 14:56:46.039799 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.039749 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 14:56:46.234073 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.234002 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:46.234549 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.234529 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:46.455832 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.455731 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T14:56:46.039796271Z","UUID":"438f35aa-c54d-4697-80b8-6c7d23ee3ac1","Handler":null,"Name":"","Endpoint":""} Jul 24 14:56:46.457492 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.457469 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 14:56:46.457628 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.457516 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 14:56:46.582457 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.582369 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:46.582951 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:46.582501 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:46.753629 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.753603 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" event={"ID":"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3","Type":"ContainerStarted","Data":"8c375aa94281abaea73f2ece854d44944769b4f16b7f6bb015d8ca09c38cd0f9"} Jul 24 14:56:46.753893 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.753877 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:46.754435 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:46.754416 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mgxft" Jul 24 14:56:47.585713 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:47.585489 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:47.585713 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:47.585705 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:47.586158 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:47.585798 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:47.586243 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:47.586162 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:47.760294 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:47.760223 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" event={"ID":"3d7dd653-a238-45b8-a1cb-53e871663fc6","Type":"ContainerStarted","Data":"33503264f2d09241d4b2f60acf047f3c6b38d848db79c00f16e47e2df340ec57"} Jul 24 14:56:47.761996 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:47.761966 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" event={"ID":"7b74e50b-a842-4cc2-9022-9ebdb3dc6bf3","Type":"ContainerStarted","Data":"4272283b25b8de69a4af535d92acc520ba971efb1173d54ead2593488eadc7b4"} Jul 24 14:56:47.781538 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:47.781491 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5k2l" podStartSLOduration=3.077136512 podStartE2EDuration="25.78147431s" totalImagesPullingTime="22.704337798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:47.781238598 +0000 UTC m=+26.728732183" watchObservedRunningTime="2026-07-24 14:56:47.78147431 +0000 UTC m=+26.728967896" Jul 24 14:56:48.582586 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:48.582551 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:48.582750 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:48.582672 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:49.582122 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:49.582102 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:49.582930 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:49.582235 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:49.582930 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:49.582277 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:49.582930 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:49.582342 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:49.769729 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:49.769505 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" event={"ID":"3d7dd653-a238-45b8-a1cb-53e871663fc6","Type":"ContainerStarted","Data":"d7f90e2f1b8bd7f841005bddc0ef9dfc91b575d2985fa98579c3e2603cf32ba0"} Jul 24 14:56:49.769859 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:49.769796 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:49.773457 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:49.770614 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:49.775362 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:49.775338 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" event={"ID":"775e609b-1986-40e4-b87b-e01a2084f7a5","Type":"ContainerStarted","Data":"de90f97991e17913b08abbddc4bc1de5eff15ce191bbe4d2d289736a019ca98a"} Jul 24 14:56:49.786591 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:49.786569 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:49.787792 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:49.787766 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:49.800665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:49.800627 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" podStartSLOduration=7.812774726 podStartE2EDuration="27.800614598s" totalImagesPullingTime="19.987839872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:56:49.800050753 +0000 UTC m=+28.747544339" watchObservedRunningTime="2026-07-24 14:56:49.800614598 +0000 UTC m=+28.748108184" Jul 24 14:56:50.582297 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:50.582273 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:50.582691 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:50.582362 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:50.779287 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:50.779256 2557 generic.go:356] "Generic (PLEG): container finished" podID="775e609b-1986-40e4-b87b-e01a2084f7a5" containerID="de90f97991e17913b08abbddc4bc1de5eff15ce191bbe4d2d289736a019ca98a" exitCode=0 Jul 24 14:56:50.779501 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:50.779345 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" event={"ID":"775e609b-1986-40e4-b87b-e01a2084f7a5","Type":"ContainerDied","Data":"de90f97991e17913b08abbddc4bc1de5eff15ce191bbe4d2d289736a019ca98a"} Jul 24 14:56:50.779501 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:50.779402 2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 24 14:56:51.583553 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:51.583524 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:51.583935 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:51.583617 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:51.583935 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:51.583627 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:51.583935 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:51.583743 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:51.750443 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:51.750405 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6gjqc"] Jul 24 14:56:51.754127 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:51.754099 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2hd8j"] Jul 24 14:56:51.754294 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:51.754239 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:51.754366 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:51.754336 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:51.754826 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:51.754797 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ghk9t"] Jul 24 14:56:51.781149 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:51.781082 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:51.781292 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:51.781169 2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 24 14:56:51.781292 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:51.781092 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:51.781292 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:51.781222 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:51.781415 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:51.781297 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:52.080366 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:52.080295 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:56:52.785524 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:52.785491 2557 generic.go:356] "Generic (PLEG): container finished" podID="775e609b-1986-40e4-b87b-e01a2084f7a5" containerID="3db244b977c64db7301c115d7d596da4e6a13947d334e39401ea3b49eabfd78f" exitCode=0 Jul 24 14:56:52.785988 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:52.785570 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" event={"ID":"775e609b-1986-40e4-b87b-e01a2084f7a5","Type":"ContainerDied","Data":"3db244b977c64db7301c115d7d596da4e6a13947d334e39401ea3b49eabfd78f"} Jul 24 14:56:53.582729 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:53.582700 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:53.582895 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:53.582740 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:53.582895 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:53.582812 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:53.582895 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:53.582828 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:53.583057 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:53.582948 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:53.583057 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:53.583039 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:53.792284 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:53.792005 2557 generic.go:356] "Generic (PLEG): container finished" podID="775e609b-1986-40e4-b87b-e01a2084f7a5" containerID="2bb047fc8dbbfb8a12d4d35bfb1118f4dc21a5a52cbf769de545ff63eb696aff" exitCode=0 Jul 24 14:56:53.792284 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:53.792078 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" event={"ID":"775e609b-1986-40e4-b87b-e01a2084f7a5","Type":"ContainerDied","Data":"2bb047fc8dbbfb8a12d4d35bfb1118f4dc21a5a52cbf769de545ff63eb696aff"} Jul 24 14:56:55.254830 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:55.254799 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:55.255283 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:55.254868 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:55.255283 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.254918 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:55.255283 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.254953 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:55.255283 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.254984 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs podName:79e24294-248d-45f7-98c3-19b260dfbe88 nodeName:}" failed. No retries permitted until 2026-07-24 14:57:27.254967321 +0000 UTC m=+66.202460905 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs") pod "network-metrics-daemon-6gjqc" (UID: "79e24294-248d-45f7-98c3-19b260dfbe88") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 14:56:55.255283 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.255000 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret podName:bd1f81ff-a1c3-4714-bbec-423bdf6e06b9 nodeName:}" failed. No retries permitted until 2026-07-24 14:57:27.254993718 +0000 UTC m=+66.202487281 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret") pod "global-pull-secret-syncer-ghk9t" (UID: "bd1f81ff-a1c3-4714-bbec-423bdf6e06b9") : object "kube-system"/"original-pull-secret" not registered Jul 24 14:56:55.355645 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:55.355604 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:55.355814 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.355779 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 14:56:55.355814 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.355802 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 14:56:55.355814 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.355814 2557 projected.go:196] Error preparing data for projected volume kube-api-access-8wjxh for pod openshift-network-diagnostics/network-check-target-2hd8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:55.355974 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.355878 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh podName:a1e6028f-648d-4a7b-9ce4-b817d1bff47b nodeName:}" failed. No retries permitted until 2026-07-24 14:57:27.355861749 +0000 UTC m=+66.303355327 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wjxh" (UniqueName: "kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh") pod "network-check-target-2hd8j" (UID: "a1e6028f-648d-4a7b-9ce4-b817d1bff47b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 14:56:55.582221 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:55.582121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:55.582391 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:55.582121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:55.582391 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.582277 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hd8j" podUID="a1e6028f-648d-4a7b-9ce4-b817d1bff47b" Jul 24 14:56:55.582391 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:55.582121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:55.582391 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.582358 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghk9t" podUID="bd1f81ff-a1c3-4714-bbec-423bdf6e06b9" Jul 24 14:56:55.582620 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:55.582411 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gjqc" podUID="79e24294-248d-45f7-98c3-19b260dfbe88" Jul 24 14:56:56.468255 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.467960 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t5h8r_a5f95e57-a279-4df4-a232-1d264a35c3fb/dns-node-resolver/0.log" Jul 24 14:56:56.874550 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.874471 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeReady" Jul 24 14:56:56.874699 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.874623 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 14:56:56.926272 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.926241 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4vlbn"] Jul 24 14:56:56.930846 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.930819 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:56.933536 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.933509 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 14:56:56.933769 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.933753 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 14:56:56.934011 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.933990 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bc4wp\"" Jul 24 14:56:56.934698 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.934671 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hx2m4"] Jul 24 14:56:56.939451 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.939422 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:56.943360 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.943220 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 14:56:56.943360 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.943325 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 14:56:56.943360 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.943336 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 14:56:56.943753 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.943734 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 14:56:56.943873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.943829 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-psqz4\"" Jul 24 14:56:56.948896 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.948874 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vlbn"] Jul 24 14:56:56.958834 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.958808 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d2f7n"] Jul 24 14:56:56.962241 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.962221 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d2f7n" Jul 24 14:56:56.963863 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.963842 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hx2m4"] Jul 24 14:56:56.964953 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.964935 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 14:56:56.965049 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.964936 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 14:56:56.966485 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.966466 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 14:56:56.966584 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.966572 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-55njp\"" Jul 24 14:56:56.975429 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:56.975399 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d2f7n"] Jul 24 14:56:57.067899 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.067627 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.067899 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.067688 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2abba8f0-87ce-4adf-b88b-29915e84a3b8-crio-socket\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.067899 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.067724 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q49c\" (UniqueName: \"kubernetes.io/projected/17960c0a-1d61-4c0f-aa87-2aed37d99494-kube-api-access-5q49c\") pod \"ingress-canary-d2f7n\" (UID: \"17960c0a-1d61-4c0f-aa87-2aed37d99494\") " pod="openshift-ingress-canary/ingress-canary-d2f7n" Jul 24 14:56:57.067899 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.067782 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-tmp-dir\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.067899 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.067808 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2abba8f0-87ce-4adf-b88b-29915e84a3b8-data-volume\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.067899 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.067862 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.068330 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.067945 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17960c0a-1d61-4c0f-aa87-2aed37d99494-cert\") pod \"ingress-canary-d2f7n\" (UID: \"17960c0a-1d61-4c0f-aa87-2aed37d99494\") " pod="openshift-ingress-canary/ingress-canary-d2f7n" Jul 24 14:56:57.068330 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.067985 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-config-volume\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.068330 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.068011 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.068330 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.068067 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzdjn\" (UniqueName: \"kubernetes.io/projected/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-kube-api-access-qzdjn\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.068330 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.068097 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8f78\" (UniqueName: \"kubernetes.io/projected/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-api-access-s8f78\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.168935 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.168891 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-tmp-dir\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.169123 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.168944 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2abba8f0-87ce-4adf-b88b-29915e84a3b8-data-volume\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.169123 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169049 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.169123 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169100 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17960c0a-1d61-4c0f-aa87-2aed37d99494-cert\") pod \"ingress-canary-d2f7n\" (UID: \"17960c0a-1d61-4c0f-aa87-2aed37d99494\") " pod="openshift-ingress-canary/ingress-canary-d2f7n" Jul 24 14:56:57.169324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169133 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-config-volume\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.169324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169151 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.169324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169171 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzdjn\" (UniqueName: \"kubernetes.io/projected/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-kube-api-access-qzdjn\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.169324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8f78\" (UniqueName: \"kubernetes.io/projected/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-api-access-s8f78\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.169324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169255 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.169324 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169296 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2abba8f0-87ce-4adf-b88b-29915e84a3b8-crio-socket\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.169548 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169327 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q49c\" (UniqueName: \"kubernetes.io/projected/17960c0a-1d61-4c0f-aa87-2aed37d99494-kube-api-access-5q49c\") pod \"ingress-canary-d2f7n\" (UID: \"17960c0a-1d61-4c0f-aa87-2aed37d99494\") " pod="openshift-ingress-canary/ingress-canary-d2f7n" Jul 24 14:56:57.169548 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.169333 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 14:56:57.169548 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169342 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2abba8f0-87ce-4adf-b88b-29915e84a3b8-data-volume\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.169548 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169356 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-tmp-dir\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.169548 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.169416 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls podName:2abba8f0-87ce-4adf-b88b-29915e84a3b8 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:57.669390103 +0000 UTC m=+36.616883684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hx2m4" (UID: "2abba8f0-87ce-4adf-b88b-29915e84a3b8") : secret "insights-runtime-extractor-tls" not found Jul 24 14:56:57.169723 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.169705 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 14:56:57.169756 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169707 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-config-volume\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.169756 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.169748 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 14:56:57.169756 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.169716 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2abba8f0-87ce-4adf-b88b-29915e84a3b8-crio-socket\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.169860 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.169782 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls podName:8a312eb8-60f9-4e4c-8409-be38cb7fbbbb nodeName:}" failed. No retries permitted until 2026-07-24 14:56:57.669763583 +0000 UTC m=+36.617257146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls") pod "dns-default-4vlbn" (UID: "8a312eb8-60f9-4e4c-8409-be38cb7fbbbb") : secret "dns-default-metrics-tls" not found Jul 24 14:56:57.169860 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.169803 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm podName:2abba8f0-87ce-4adf-b88b-29915e84a3b8 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:57.66979279 +0000 UTC m=+36.617286371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hx2m4" (UID: "2abba8f0-87ce-4adf-b88b-29915e84a3b8") : configmap "kube-rbac-proxy" not found Jul 24 14:56:57.173937 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.173918 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17960c0a-1d61-4c0f-aa87-2aed37d99494-cert\") pod \"ingress-canary-d2f7n\" (UID: \"17960c0a-1d61-4c0f-aa87-2aed37d99494\") " pod="openshift-ingress-canary/ingress-canary-d2f7n" Jul 24 14:56:57.183253 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.183070 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q49c\" (UniqueName: \"kubernetes.io/projected/17960c0a-1d61-4c0f-aa87-2aed37d99494-kube-api-access-5q49c\") pod \"ingress-canary-d2f7n\" (UID: \"17960c0a-1d61-4c0f-aa87-2aed37d99494\") " pod="openshift-ingress-canary/ingress-canary-d2f7n" Jul 24 14:56:57.183539 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.183514 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzdjn\" (UniqueName: \"kubernetes.io/projected/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-kube-api-access-qzdjn\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.184637 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.184616 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8f78\" (UniqueName: \"kubernetes.io/projected/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-api-access-s8f78\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.272951 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.272910 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d2f7n" Jul 24 14:56:57.586058 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.585680 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:56:57.586058 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.585723 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:56:57.586058 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.585785 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:56:57.588960 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.588939 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 14:56:57.589098 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.588988 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 14:56:57.589098 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.589018 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 14:56:57.589098 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.589020 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 14:56:57.589234 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.589105 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5tbx2\"" Jul 24 14:56:57.590323 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.590298 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j76t9\"" Jul 24 14:56:57.643144 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.643111 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s2rlr_668e6b2b-8552-4dc8-a40d-3d7961a9b066/node-ca/0.log" Jul 24 14:56:57.673457 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.673424 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.673610 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.673472 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:57.673610 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:57.673520 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:57.673610 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.673600 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 14:56:57.673754 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.673607 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 14:56:57.673754 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.673648 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 14:56:57.673754 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.673660 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm podName:2abba8f0-87ce-4adf-b88b-29915e84a3b8 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:58.673639765 +0000 UTC m=+37.621133329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hx2m4" (UID: "2abba8f0-87ce-4adf-b88b-29915e84a3b8") : configmap "kube-rbac-proxy" not found Jul 24 14:56:57.673754 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.673678 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls podName:2abba8f0-87ce-4adf-b88b-29915e84a3b8 nodeName:}" failed. No retries permitted until 2026-07-24 14:56:58.673665363 +0000 UTC m=+37.621158930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hx2m4" (UID: "2abba8f0-87ce-4adf-b88b-29915e84a3b8") : secret "insights-runtime-extractor-tls" not found Jul 24 14:56:57.673754 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:57.673695 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls podName:8a312eb8-60f9-4e4c-8409-be38cb7fbbbb nodeName:}" failed. No retries permitted until 2026-07-24 14:56:58.673687546 +0000 UTC m=+37.621181109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls") pod "dns-default-4vlbn" (UID: "8a312eb8-60f9-4e4c-8409-be38cb7fbbbb") : secret "dns-default-metrics-tls" not found Jul 24 14:56:58.207489 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:58.207343 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d2f7n"] Jul 24 14:56:58.211052 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:56:58.211019 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17960c0a_1d61_4c0f_aa87_2aed37d99494.slice/crio-02ead48c575eed5d47a412f8a7a45aebbc441fbb00374d494055d7de90ea17d8 WatchSource:0}: Error finding container 02ead48c575eed5d47a412f8a7a45aebbc441fbb00374d494055d7de90ea17d8: Status 404 returned error can't find the container with id 02ead48c575eed5d47a412f8a7a45aebbc441fbb00374d494055d7de90ea17d8 Jul 24 14:56:58.681558 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:58.681531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:58.681995 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:58.681594 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:56:58.681995 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:58.681621 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:56:58.681995 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:58.681658 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 14:56:58.681995 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:58.681707 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 14:56:58.681995 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:58.681736 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm podName:2abba8f0-87ce-4adf-b88b-29915e84a3b8 nodeName:}" failed. No retries permitted until 2026-07-24 14:57:00.681714588 +0000 UTC m=+39.629208158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hx2m4" (UID: "2abba8f0-87ce-4adf-b88b-29915e84a3b8") : configmap "kube-rbac-proxy" not found Jul 24 14:56:58.681995 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:58.681753 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 14:56:58.681995 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:58.681756 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls podName:8a312eb8-60f9-4e4c-8409-be38cb7fbbbb nodeName:}" failed. No retries permitted until 2026-07-24 14:57:00.681747435 +0000 UTC m=+39.629241001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls") pod "dns-default-4vlbn" (UID: "8a312eb8-60f9-4e4c-8409-be38cb7fbbbb") : secret "dns-default-metrics-tls" not found Jul 24 14:56:58.681995 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:56:58.681792 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls podName:2abba8f0-87ce-4adf-b88b-29915e84a3b8 nodeName:}" failed. No retries permitted until 2026-07-24 14:57:00.681784332 +0000 UTC m=+39.629277896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hx2m4" (UID: "2abba8f0-87ce-4adf-b88b-29915e84a3b8") : secret "insights-runtime-extractor-tls" not found Jul 24 14:56:58.806432 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:58.806389 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d2f7n" event={"ID":"17960c0a-1d61-4c0f-aa87-2aed37d99494","Type":"ContainerStarted","Data":"02ead48c575eed5d47a412f8a7a45aebbc441fbb00374d494055d7de90ea17d8"} Jul 24 14:56:58.809566 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:58.809536 2557 generic.go:356] "Generic (PLEG): container finished" podID="775e609b-1986-40e4-b87b-e01a2084f7a5" containerID="6e82569d756a9e538c361babf77ff83ab35b8a51952daff441c086689cf1c827" exitCode=0 Jul 24 14:56:58.809727 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:58.809581 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" event={"ID":"775e609b-1986-40e4-b87b-e01a2084f7a5","Type":"ContainerDied","Data":"6e82569d756a9e538c361babf77ff83ab35b8a51952daff441c086689cf1c827"} Jul 24 14:56:59.816210 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:59.816155 2557 generic.go:356] "Generic (PLEG): container finished" podID="775e609b-1986-40e4-b87b-e01a2084f7a5" containerID="ac31ed728a07e13de7d47aba4baeb50d1ba43c0d70fe22123b7aec151d6f02c3" exitCode=0 Jul 24 14:56:59.816647 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:56:59.816217 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" event={"ID":"775e609b-1986-40e4-b87b-e01a2084f7a5","Type":"ContainerDied","Data":"ac31ed728a07e13de7d47aba4baeb50d1ba43c0d70fe22123b7aec151d6f02c3"} Jul 24 14:57:00.698540 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:00.698503 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:57:00.698692 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:00.698552 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:57:00.698692 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:00.698597 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:57:00.698692 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:00.698661 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 14:57:00.698797 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:00.698714 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 14:57:00.698797 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:00.698730 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls podName:2abba8f0-87ce-4adf-b88b-29915e84a3b8 nodeName:}" failed. No retries permitted until 2026-07-24 14:57:04.698711243 +0000 UTC m=+43.646204809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hx2m4" (UID: "2abba8f0-87ce-4adf-b88b-29915e84a3b8") : secret "insights-runtime-extractor-tls" not found Jul 24 14:57:00.698797 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:00.698769 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls podName:8a312eb8-60f9-4e4c-8409-be38cb7fbbbb nodeName:}" failed. No retries permitted until 2026-07-24 14:57:04.698753727 +0000 UTC m=+43.646247292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls") pod "dns-default-4vlbn" (UID: "8a312eb8-60f9-4e4c-8409-be38cb7fbbbb") : secret "dns-default-metrics-tls" not found Jul 24 14:57:00.699213 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:00.699165 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abba8f0-87ce-4adf-b88b-29915e84a3b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:57:00.823131 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:00.823095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" event={"ID":"775e609b-1986-40e4-b87b-e01a2084f7a5","Type":"ContainerStarted","Data":"40962b526f73221d0d164d2de194e17a297b897938ec00aeae3c8878abb6410b"} Jul 24 14:57:00.851918 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:00.851857 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jd6nn" podStartSLOduration=7.802850489 podStartE2EDuration="38.851837952s" totalImagesPullingTime="31.048987463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:00.851481561 +0000 UTC m=+39.798975169" watchObservedRunningTime="2026-07-24 14:57:00.851837952 +0000 UTC m=+39.799331539" Jul 24 14:57:01.827235 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:01.826948 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d2f7n" event={"ID":"17960c0a-1d61-4c0f-aa87-2aed37d99494","Type":"ContainerStarted","Data":"c693a77d8781eaca157d2327619ba2c19d4972374257da99184b8fb549959faa"} Jul 24 14:57:01.844179 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:01.844133 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d2f7n" podStartSLOduration=3.32252098 podStartE2EDuration="5.844117656s" totalImagesPullingTime="2.521596676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:01.843489454 +0000 UTC m=+40.790983044" watchObservedRunningTime="2026-07-24 14:57:01.844117656 +0000 UTC m=+40.791611242" Jul 24 14:57:04.729873 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:04.729837 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:57:04.730272 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:04.729884 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:57:04.730272 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:04.730001 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 14:57:04.730272 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:04.730064 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls podName:2abba8f0-87ce-4adf-b88b-29915e84a3b8 nodeName:}" failed. No retries permitted until 2026-07-24 14:57:12.730049619 +0000 UTC m=+51.677543183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hx2m4" (UID: "2abba8f0-87ce-4adf-b88b-29915e84a3b8") : secret "insights-runtime-extractor-tls" not found Jul 24 14:57:04.730272 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:04.730001 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 14:57:04.730272 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:04.730121 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls podName:8a312eb8-60f9-4e4c-8409-be38cb7fbbbb nodeName:}" failed. No retries permitted until 2026-07-24 14:57:12.73011135 +0000 UTC m=+51.677604919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls") pod "dns-default-4vlbn" (UID: "8a312eb8-60f9-4e4c-8409-be38cb7fbbbb") : secret "dns-default-metrics-tls" not found Jul 24 14:57:12.791711 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:12.791671 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:57:12.791711 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:12.791716 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:57:12.795644 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:12.795619 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a312eb8-60f9-4e4c-8409-be38cb7fbbbb-metrics-tls\") pod \"dns-default-4vlbn\" (UID: \"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb\") " pod="openshift-dns/dns-default-4vlbn" Jul 24 14:57:12.795756 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:12.795654 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abba8f0-87ce-4adf-b88b-29915e84a3b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hx2m4\" (UID: \"2abba8f0-87ce-4adf-b88b-29915e84a3b8\") " pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:57:12.845275 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:12.845244 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vlbn" Jul 24 14:57:12.851365 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:12.851335 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hx2m4" Jul 24 14:57:12.992519 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:12.992488 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vlbn"] Jul 24 14:57:12.995117 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:12.995087 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a312eb8_60f9_4e4c_8409_be38cb7fbbbb.slice/crio-e86ada83ca952ac7b00d9d618267886c3c4fdfa6a9ce502810eec43777752fb4 WatchSource:0}: Error finding container e86ada83ca952ac7b00d9d618267886c3c4fdfa6a9ce502810eec43777752fb4: Status 404 returned error can't find the container with id e86ada83ca952ac7b00d9d618267886c3c4fdfa6a9ce502810eec43777752fb4 Jul 24 14:57:13.010293 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:13.010268 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hx2m4"] Jul 24 14:57:13.025090 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:13.025057 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abba8f0_87ce_4adf_b88b_29915e84a3b8.slice/crio-6c842814abcff350678080cd82049eb16fed7b4d1355511c78dd43b44c4a5981 WatchSource:0}: Error finding container 6c842814abcff350678080cd82049eb16fed7b4d1355511c78dd43b44c4a5981: Status 404 returned error can't find the container with id 6c842814abcff350678080cd82049eb16fed7b4d1355511c78dd43b44c4a5981 Jul 24 14:57:13.858030 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:13.857977 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hx2m4" event={"ID":"2abba8f0-87ce-4adf-b88b-29915e84a3b8","Type":"ContainerStarted","Data":"6c842814abcff350678080cd82049eb16fed7b4d1355511c78dd43b44c4a5981"} Jul 24 14:57:13.859436 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:13.859370 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vlbn" event={"ID":"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb","Type":"ContainerStarted","Data":"e86ada83ca952ac7b00d9d618267886c3c4fdfa6a9ce502810eec43777752fb4"} Jul 24 14:57:15.867156 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:15.867062 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vlbn" event={"ID":"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb","Type":"ContainerStarted","Data":"9fec195c52e2ebf8ba4e4c678543896477a012b87f6d146889f86b7ca6ecd524"} Jul 24 14:57:15.867156 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:15.867137 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vlbn" event={"ID":"8a312eb8-60f9-4e4c-8409-be38cb7fbbbb","Type":"ContainerStarted","Data":"4ff3c7c7752da00c1a6fb4a3fdb6906fc8be095b03f27dd8737bf02fcabbe8a2"} Jul 24 14:57:15.867676 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:15.867248 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4vlbn" Jul 24 14:57:15.886721 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:15.886669 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4vlbn" podStartSLOduration=17.463913387 podStartE2EDuration="19.886649117s" totalImagesPullingTime="2.42273573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:15.886023741 +0000 UTC m=+54.833517328" watchObservedRunningTime="2026-07-24 14:57:15.886649117 +0000 UTC m=+54.834142704" Jul 24 14:57:18.876532 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:18.876488 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hx2m4" event={"ID":"2abba8f0-87ce-4adf-b88b-29915e84a3b8","Type":"ContainerStarted","Data":"c5bc401c1948bfefaaa521a3eca3f06e44093c124267f08b7eed4d1aad145a86"} Jul 24 14:57:20.308491 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.307632 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn"] Jul 24 14:57:20.321853 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.321826 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" Jul 24 14:57:20.325923 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.325896 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 14:57:20.327048 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.327026 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 14:57:20.327343 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.327328 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pqkr4\"" Jul 24 14:57:20.327459 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.327365 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 14:57:20.328560 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.328382 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 14:57:20.334661 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.334638 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94"] Jul 24 14:57:20.339644 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.339624 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.346742 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.346715 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn"] Jul 24 14:57:20.352543 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.352367 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09e2df83-789d-47db-949b-be9093d77094-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7676847cfd-4qwzn\" (UID: \"09e2df83-789d-47db-949b-be9093d77094\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" Jul 24 14:57:20.352543 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.352473 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lzt\" (UniqueName: \"kubernetes.io/projected/09e2df83-789d-47db-949b-be9093d77094-kube-api-access-q9lzt\") pod \"managed-serviceaccount-addon-agent-7676847cfd-4qwzn\" (UID: \"09e2df83-789d-47db-949b-be9093d77094\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" Jul 24 14:57:20.352824 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.352804 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 14:57:20.353014 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.352989 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 14:57:20.359128 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.359101 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 14:57:20.359897 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.359878 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 14:57:20.409293 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.409145 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94"] Jul 24 14:57:20.452784 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.452748 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-hub\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.452990 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.452800 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09e2df83-789d-47db-949b-be9093d77094-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7676847cfd-4qwzn\" (UID: \"09e2df83-789d-47db-949b-be9093d77094\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" Jul 24 14:57:20.452990 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.452830 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h799b\" (UniqueName: \"kubernetes.io/projected/cc3393be-438e-4136-8d1a-b0e1063548d1-kube-api-access-h799b\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.452990 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.452854 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3393be-438e-4136-8d1a-b0e1063548d1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.452990 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.452899 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-ca\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.452990 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.452923 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.452990 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.452968 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lzt\" (UniqueName: \"kubernetes.io/projected/09e2df83-789d-47db-949b-be9093d77094-kube-api-access-q9lzt\") pod \"managed-serviceaccount-addon-agent-7676847cfd-4qwzn\" (UID: \"09e2df83-789d-47db-949b-be9093d77094\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" Jul 24 14:57:20.453412 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.452994 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.456163 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.456138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09e2df83-789d-47db-949b-be9093d77094-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7676847cfd-4qwzn\" (UID: \"09e2df83-789d-47db-949b-be9093d77094\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" Jul 24 14:57:20.486474 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.486437 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lzt\" (UniqueName: \"kubernetes.io/projected/09e2df83-789d-47db-949b-be9093d77094-kube-api-access-q9lzt\") pod \"managed-serviceaccount-addon-agent-7676847cfd-4qwzn\" (UID: \"09e2df83-789d-47db-949b-be9093d77094\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" Jul 24 14:57:20.554020 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.553984 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.554020 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.554025 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-hub\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.554277 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.554062 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h799b\" (UniqueName: \"kubernetes.io/projected/cc3393be-438e-4136-8d1a-b0e1063548d1-kube-api-access-h799b\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.554277 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.554090 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3393be-438e-4136-8d1a-b0e1063548d1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.554277 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.554126 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-ca\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.554277 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.554142 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.555107 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.555056 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3393be-438e-4136-8d1a-b0e1063548d1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.557334 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.557304 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-hub\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.557451 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.557348 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-ca\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.557451 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.557418 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.557555 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.557469 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cc3393be-438e-4136-8d1a-b0e1063548d1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.565750 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.565699 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h799b\" (UniqueName: \"kubernetes.io/projected/cc3393be-438e-4136-8d1a-b0e1063548d1-kube-api-access-h799b\") pod \"cluster-proxy-proxy-agent-7c86ff6fd8-zxk94\" (UID: \"cc3393be-438e-4136-8d1a-b0e1063548d1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:20.642936 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.642903 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" Jul 24 14:57:20.653091 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:20.653061 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" Jul 24 14:57:22.146549 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:22.146452 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94"] Jul 24 14:57:22.149034 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:22.148997 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc3393be_438e_4136_8d1a_b0e1063548d1.slice/crio-7daa086cca8ac18d3e4f2b3613327f9fb723ab6a0bdcb7e8095a1b5f2631fde0 WatchSource:0}: Error finding container 7daa086cca8ac18d3e4f2b3613327f9fb723ab6a0bdcb7e8095a1b5f2631fde0: Status 404 returned error can't find the container with id 7daa086cca8ac18d3e4f2b3613327f9fb723ab6a0bdcb7e8095a1b5f2631fde0 Jul 24 14:57:22.158556 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:22.158531 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn"] Jul 24 14:57:22.161268 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:22.161218 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09e2df83_789d_47db_949b_be9093d77094.slice/crio-f9ff610b571db8ab7660ce4849e4945e5abfb8fda6c3890732ab158b39174787 WatchSource:0}: Error finding container f9ff610b571db8ab7660ce4849e4945e5abfb8fda6c3890732ab158b39174787: Status 404 returned error can't find the container with id f9ff610b571db8ab7660ce4849e4945e5abfb8fda6c3890732ab158b39174787 Jul 24 14:57:22.801386 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:22.801353 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vt4v8" Jul 24 14:57:22.895345 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:22.895300 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hx2m4" event={"ID":"2abba8f0-87ce-4adf-b88b-29915e84a3b8","Type":"ContainerStarted","Data":"b272f10885e130311f30595a1ed1b1bff8e1277fbe6c7767610c997770d066ab"} Jul 24 14:57:22.898033 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:22.898002 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" event={"ID":"cc3393be-438e-4136-8d1a-b0e1063548d1","Type":"ContainerStarted","Data":"7daa086cca8ac18d3e4f2b3613327f9fb723ab6a0bdcb7e8095a1b5f2631fde0"} Jul 24 14:57:22.900690 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:22.900661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" event={"ID":"09e2df83-789d-47db-949b-be9093d77094","Type":"ContainerStarted","Data":"f9ff610b571db8ab7660ce4849e4945e5abfb8fda6c3890732ab158b39174787"} Jul 24 14:57:24.779710 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:24.779667 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9"] Jul 24 14:57:24.783553 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:24.782834 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" Jul 24 14:57:24.786124 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:24.786102 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jfxcd\"" Jul 24 14:57:24.786344 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:24.786327 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 24 14:57:24.793664 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:24.793638 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9"] Jul 24 14:57:24.897047 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:24.896997 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddaa3b2c-a46c-46d2-a805-b5f5b7a55707-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8ntc9\" (UID: \"ddaa3b2c-a46c-46d2-a805-b5f5b7a55707\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" Jul 24 14:57:24.998532 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:24.998497 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddaa3b2c-a46c-46d2-a805-b5f5b7a55707-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8ntc9\" (UID: \"ddaa3b2c-a46c-46d2-a805-b5f5b7a55707\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" Jul 24 14:57:24.998750 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:24.998682 2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 24 14:57:24.998902 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:24.998766 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddaa3b2c-a46c-46d2-a805-b5f5b7a55707-tls-certificates podName:ddaa3b2c-a46c-46d2-a805-b5f5b7a55707 nodeName:}" failed. No retries permitted until 2026-07-24 14:57:25.498748362 +0000 UTC m=+64.446241926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ddaa3b2c-a46c-46d2-a805-b5f5b7a55707-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" (UID: "ddaa3b2c-a46c-46d2-a805-b5f5b7a55707") : secret "prometheus-operator-admission-webhook-tls" not found Jul 24 14:57:25.503120 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:25.503085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddaa3b2c-a46c-46d2-a805-b5f5b7a55707-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8ntc9\" (UID: \"ddaa3b2c-a46c-46d2-a805-b5f5b7a55707\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" Jul 24 14:57:25.505903 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:25.505873 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddaa3b2c-a46c-46d2-a805-b5f5b7a55707-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8ntc9\" (UID: \"ddaa3b2c-a46c-46d2-a805-b5f5b7a55707\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" Jul 24 14:57:25.696236 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:25.696195 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" Jul 24 14:57:25.873004 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:25.872913 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4vlbn" Jul 24 14:57:25.914003 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:25.913970 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" event={"ID":"09e2df83-789d-47db-949b-be9093d77094","Type":"ContainerStarted","Data":"0e1fcafd0dc2bcc31bee8c0a071ed6bc45497dfc71c239178e2fb98443e49f8f"} Jul 24 14:57:25.931897 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:25.931846 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7676847cfd-4qwzn" podStartSLOduration=3.020183082 podStartE2EDuration="5.931829344s" totalImagesPullingTime="2.911646262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:25.931049101 +0000 UTC m=+64.878542692" watchObservedRunningTime="2026-07-24 14:57:25.931829344 +0000 UTC m=+64.879322930" Jul 24 14:57:27.318574 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.318527 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:57:27.319037 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.318622 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:57:27.321626 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.321594 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 14:57:27.321939 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.321918 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 14:57:27.331779 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.331746 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e24294-248d-45f7-98c3-19b260dfbe88-metrics-certs\") pod \"network-metrics-daemon-6gjqc\" (UID: \"79e24294-248d-45f7-98c3-19b260dfbe88\") " pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:57:27.331919 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.331812 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd1f81ff-a1c3-4714-bbec-423bdf6e06b9-original-pull-secret\") pod \"global-pull-secret-syncer-ghk9t\" (UID: \"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9\") " pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:57:27.418964 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.418931 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:57:27.422131 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.422108 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 14:57:27.433372 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.433340 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 14:57:27.442296 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.442273 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjxh\" (UniqueName: \"kubernetes.io/projected/a1e6028f-648d-4a7b-9ce4-b817d1bff47b-kube-api-access-8wjxh\") pod \"network-check-target-2hd8j\" (UID: \"a1e6028f-648d-4a7b-9ce4-b817d1bff47b\") " pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:57:27.597835 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.597766 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghk9t" Jul 24 14:57:27.607896 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.607870 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5tbx2\"" Jul 24 14:57:27.613618 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.613590 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j76t9\"" Jul 24 14:57:27.615673 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.615648 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gjqc" Jul 24 14:57:27.622166 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:27.622147 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:57:28.734826 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:28.734799 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2hd8j"] Jul 24 14:57:28.754733 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:28.754690 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ghk9t"] Jul 24 14:57:28.758339 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:28.758309 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1f81ff_a1c3_4714_bbec_423bdf6e06b9.slice/crio-8f78857d8b2ff067f383587a8c8a57b0b6a72c3d714fb2327cf5707fc9e4c2cd WatchSource:0}: Error finding container 8f78857d8b2ff067f383587a8c8a57b0b6a72c3d714fb2327cf5707fc9e4c2cd: Status 404 returned error can't find the container with id 8f78857d8b2ff067f383587a8c8a57b0b6a72c3d714fb2327cf5707fc9e4c2cd Jul 24 14:57:28.924911 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:28.924879 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hx2m4" event={"ID":"2abba8f0-87ce-4adf-b88b-29915e84a3b8","Type":"ContainerStarted","Data":"b2db9f3f1d762f47fa61e7f9f660f9f1cb0bb897b73ca30f40b1c47fc5e8b619"} Jul 24 14:57:28.926669 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:28.926640 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" event={"ID":"cc3393be-438e-4136-8d1a-b0e1063548d1","Type":"ContainerStarted","Data":"ee980271b4020695fdd54cae225aab9ad85a45a7e41e669c2e532c3cf82b4185"} Jul 24 14:57:28.926770 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:28.926680 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" event={"ID":"cc3393be-438e-4136-8d1a-b0e1063548d1","Type":"ContainerStarted","Data":"b78d72e0475c322edf4a0a2ee07526243e34bd91fa595ae2653db636c05f8b61"} Jul 24 14:57:28.927640 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:28.927619 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2hd8j" event={"ID":"a1e6028f-648d-4a7b-9ce4-b817d1bff47b","Type":"ContainerStarted","Data":"b6fa482db55d3277c221ca7447de8604dfeeaf80e186ca23c1155eb471b0bed5"} Jul 24 14:57:28.928557 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:28.928538 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ghk9t" event={"ID":"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9","Type":"ContainerStarted","Data":"8f78857d8b2ff067f383587a8c8a57b0b6a72c3d714fb2327cf5707fc9e4c2cd"} Jul 24 14:57:28.971219 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:28.971150 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hx2m4" podStartSLOduration=17.523140912 podStartE2EDuration="32.971132609s" totalImagesPullingTime="15.447991697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:28.970324555 +0000 UTC m=+67.917818170" watchObservedRunningTime="2026-07-24 14:57:28.971132609 +0000 UTC m=+67.918626194" Jul 24 14:57:29.004779 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:29.004750 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9"] Jul 24 14:57:29.008829 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:29.008799 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaa3b2c_a46c_46d2_a805_b5f5b7a55707.slice/crio-935d5ccbc4ec2bb5e486dcda2275350922fadf675081987c38d115f74bc4c341 WatchSource:0}: Error finding container 935d5ccbc4ec2bb5e486dcda2275350922fadf675081987c38d115f74bc4c341: Status 404 returned error can't find the container with id 935d5ccbc4ec2bb5e486dcda2275350922fadf675081987c38d115f74bc4c341 Jul 24 14:57:29.013433 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:29.013411 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6gjqc"] Jul 24 14:57:29.015775 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:29.015753 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e24294_248d_45f7_98c3_19b260dfbe88.slice/crio-86985165f458d8be451d6342c21eef7604616b81947642f51d9eaf690b4fc679 WatchSource:0}: Error finding container 86985165f458d8be451d6342c21eef7604616b81947642f51d9eaf690b4fc679: Status 404 returned error can't find the container with id 86985165f458d8be451d6342c21eef7604616b81947642f51d9eaf690b4fc679 Jul 24 14:57:29.940423 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:29.939917 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" event={"ID":"cc3393be-438e-4136-8d1a-b0e1063548d1","Type":"ContainerStarted","Data":"0e8c0e8d80ff39525f857759c254bf30301a61ce06f935e51cc3401966f84f73"} Jul 24 14:57:29.942254 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:29.942227 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" event={"ID":"ddaa3b2c-a46c-46d2-a805-b5f5b7a55707","Type":"ContainerStarted","Data":"935d5ccbc4ec2bb5e486dcda2275350922fadf675081987c38d115f74bc4c341"} Jul 24 14:57:29.945398 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:29.945366 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6gjqc" event={"ID":"79e24294-248d-45f7-98c3-19b260dfbe88","Type":"ContainerStarted","Data":"86985165f458d8be451d6342c21eef7604616b81947642f51d9eaf690b4fc679"} Jul 24 14:57:29.962314 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:29.962085 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c86ff6fd8-zxk94" podStartSLOduration=3.539565982 podStartE2EDuration="9.962067041s" totalImagesPullingTime="6.422501059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:29.960941048 +0000 UTC m=+68.908434659" watchObservedRunningTime="2026-07-24 14:57:29.962067041 +0000 UTC m=+68.909560629" Jul 24 14:57:30.951421 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:30.950932 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6gjqc" event={"ID":"79e24294-248d-45f7-98c3-19b260dfbe88","Type":"ContainerStarted","Data":"883074800b6919c1c01363ddbf0139f6e242ae39d020cb707368877d9ecd75be"} Jul 24 14:57:30.951421 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:30.951000 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6gjqc" event={"ID":"79e24294-248d-45f7-98c3-19b260dfbe88","Type":"ContainerStarted","Data":"5302ac753ac0b9df837e8a60a54ea825198e9cdc60363fa608ed6a48c265722a"} Jul 24 14:57:30.953083 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:30.953035 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" event={"ID":"ddaa3b2c-a46c-46d2-a805-b5f5b7a55707","Type":"ContainerStarted","Data":"752d55a8a9dc90438dace3dd16a2cd539e9d63f5ebd46e5fdb9b5499c4f8c640"} Jul 24 14:57:30.953225 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:30.953122 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" Jul 24 14:57:30.959586 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:30.959570 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" Jul 24 14:57:30.973974 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:30.973936 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6gjqc" podStartSLOduration=67.495358513 podStartE2EDuration="1m8.973921016s" totalImagesPullingTime="1.478562503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:30.973392639 +0000 UTC m=+69.920886233" watchObservedRunningTime="2026-07-24 14:57:30.973921016 +0000 UTC m=+69.921414601" Jul 24 14:57:30.991817 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:30.991769 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8ntc9" podStartSLOduration=5.50346174 podStartE2EDuration="6.991754389s" totalImagesPullingTime="1.488292649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:30.991463158 +0000 UTC m=+69.938956745" watchObservedRunningTime="2026-07-24 14:57:30.991754389 +0000 UTC m=+69.939247975" Jul 24 14:57:34.817358 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.817320 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-854b698b9d-mjswt"] Jul 24 14:57:34.820136 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.820118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.822740 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.822706 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 14:57:34.822874 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.822850 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 14:57:34.822874 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.822859 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 14:57:34.823009 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.822848 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 14:57:34.823009 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.822985 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 14:57:34.824045 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.824026 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 14:57:34.824147 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.824059 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 14:57:34.824147 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.824097 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gfgt9\"" Jul 24 14:57:34.836355 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.836333 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854b698b9d-mjswt"] Jul 24 14:57:34.873259 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.873228 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-serving-cert\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.873259 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.873257 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-config\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.873466 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.873279 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-oauth-config\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.873466 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.873295 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-service-ca\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.873466 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.873315 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-oauth-serving-cert\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.873466 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.873392 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxv8\" (UniqueName: \"kubernetes.io/projected/cb3b5c4a-944f-40bc-8602-7f5832083cd8-kube-api-access-rzxv8\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.969414 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.969383 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2hd8j" event={"ID":"a1e6028f-648d-4a7b-9ce4-b817d1bff47b","Type":"ContainerStarted","Data":"1a71ceae73b2982108a53a5b1038af1c6cf14ef585022c098dfa076682f65b33"} Jul 24 14:57:34.969544 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.969481 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:57:34.970828 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.970805 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ghk9t" event={"ID":"bd1f81ff-a1c3-4714-bbec-423bdf6e06b9","Type":"ContainerStarted","Data":"7e30451a42a896454215eb20a4c6754a983646b66d13bf715fa41f7e45e48d19"} Jul 24 14:57:34.973703 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.973686 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-oauth-config\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.973810 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.973716 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-service-ca\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.973810 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.973744 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-oauth-serving-cert\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.973810 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.973784 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxv8\" (UniqueName: \"kubernetes.io/projected/cb3b5c4a-944f-40bc-8602-7f5832083cd8-kube-api-access-rzxv8\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.973950 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.973839 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-serving-cert\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.973950 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.973867 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-config\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.974562 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.974540 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-service-ca\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.974686 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.974540 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-oauth-serving-cert\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.974686 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.974603 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-config\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.976206 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.976173 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-oauth-config\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.976321 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.976307 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-serving-cert\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.982676 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.982655 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxv8\" (UniqueName: \"kubernetes.io/projected/cb3b5c4a-944f-40bc-8602-7f5832083cd8-kube-api-access-rzxv8\") pod \"console-854b698b9d-mjswt\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:34.985987 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:34.985943 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2hd8j" podStartSLOduration=67.790152438 podStartE2EDuration="1m12.985928962s" totalImagesPullingTime="5.195776524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:34.985745891 +0000 UTC m=+73.933239490" watchObservedRunningTime="2026-07-24 14:57:34.985928962 +0000 UTC m=+73.933422547" Jul 24 14:57:35.002588 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:35.002543 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ghk9t" podStartSLOduration=67.82015866 podStartE2EDuration="1m13.002531055s" totalImagesPullingTime="5.182372395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:35.00183298 +0000 UTC m=+73.949326579" watchObservedRunningTime="2026-07-24 14:57:35.002531055 +0000 UTC m=+73.950024640" Jul 24 14:57:35.129037 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:35.128943 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:35.259521 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:35.259497 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854b698b9d-mjswt"] Jul 24 14:57:35.261984 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:35.261958 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3b5c4a_944f_40bc_8602_7f5832083cd8.slice/crio-10c056ed3de9874795c7d54a85dcd5d817030595063de58e8fa1d2bde2993f98 WatchSource:0}: Error finding container 10c056ed3de9874795c7d54a85dcd5d817030595063de58e8fa1d2bde2993f98: Status 404 returned error can't find the container with id 10c056ed3de9874795c7d54a85dcd5d817030595063de58e8fa1d2bde2993f98 Jul 24 14:57:35.976166 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:35.976128 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854b698b9d-mjswt" event={"ID":"cb3b5c4a-944f-40bc-8602-7f5832083cd8","Type":"ContainerStarted","Data":"10c056ed3de9874795c7d54a85dcd5d817030595063de58e8fa1d2bde2993f98"} Jul 24 14:57:37.618917 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.618883 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cn668"] Jul 24 14:57:37.622311 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.622290 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.625464 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.625439 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 14:57:37.625602 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.625568 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 14:57:37.626063 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.626046 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 14:57:37.626664 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.626500 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 14:57:37.626664 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.626561 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 14:57:37.626664 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.626592 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89sm5\"" Jul 24 14:57:37.626664 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.626621 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 14:57:37.699724 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.699685 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d640e5ab-e9d6-43c6-8081-468e57639679-metrics-client-ca\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.699910 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.699732 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-textfile\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.699910 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.699757 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvsd\" (UniqueName: \"kubernetes.io/projected/d640e5ab-e9d6-43c6-8081-468e57639679-kube-api-access-klvsd\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.699910 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.699830 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.700044 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.699944 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d640e5ab-e9d6-43c6-8081-468e57639679-root\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.700044 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.699988 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-wtmp\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.700179 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.700048 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-tls\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.700179 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.700084 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-accelerators-collector-config\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.700179 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.700125 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d640e5ab-e9d6-43c6-8081-468e57639679-sys\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.801528 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.801497 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d640e5ab-e9d6-43c6-8081-468e57639679-metrics-client-ca\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.801528 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.801538 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-textfile\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.801772 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.801563 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvsd\" (UniqueName: \"kubernetes.io/projected/d640e5ab-e9d6-43c6-8081-468e57639679-kube-api-access-klvsd\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.801772 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.801599 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.801772 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.801650 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d640e5ab-e9d6-43c6-8081-468e57639679-root\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.801772 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.801715 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d640e5ab-e9d6-43c6-8081-468e57639679-root\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.801956 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.801911 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-textfile\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.802015 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.801959 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-wtmp\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.802015 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.801985 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-tls\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.802015 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.802009 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-accelerators-collector-config\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.802157 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.802045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d640e5ab-e9d6-43c6-8081-468e57639679-sys\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.802157 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.802113 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d640e5ab-e9d6-43c6-8081-468e57639679-sys\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.802157 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.802115 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-wtmp\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.802157 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:37.802136 2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 14:57:37.802157 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.802149 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d640e5ab-e9d6-43c6-8081-468e57639679-metrics-client-ca\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.802415 ip-10-0-139-100 kubenswrapper[2557]: E0724 14:57:37.802228 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-tls podName:d640e5ab-e9d6-43c6-8081-468e57639679 nodeName:}" failed. No retries permitted until 2026-07-24 14:57:38.30221186 +0000 UTC m=+77.249705424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-tls") pod "node-exporter-cn668" (UID: "d640e5ab-e9d6-43c6-8081-468e57639679") : secret "node-exporter-tls" not found Jul 24 14:57:37.802606 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.802559 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-accelerators-collector-config\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.804694 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.804670 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:37.813930 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:37.813148 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvsd\" (UniqueName: \"kubernetes.io/projected/d640e5ab-e9d6-43c6-8081-468e57639679-kube-api-access-klvsd\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:38.306210 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:38.306150 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-tls\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:38.308524 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:38.308500 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d640e5ab-e9d6-43c6-8081-468e57639679-node-exporter-tls\") pod \"node-exporter-cn668\" (UID: \"d640e5ab-e9d6-43c6-8081-468e57639679\") " pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:38.535143 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:38.535113 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cn668" Jul 24 14:57:38.544969 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:38.544945 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd640e5ab_e9d6_43c6_8081_468e57639679.slice/crio-1f16405fda3a12e648cbc3e34137fd68bc5033aaa5d9c694148c6165cb84d764 WatchSource:0}: Error finding container 1f16405fda3a12e648cbc3e34137fd68bc5033aaa5d9c694148c6165cb84d764: Status 404 returned error can't find the container with id 1f16405fda3a12e648cbc3e34137fd68bc5033aaa5d9c694148c6165cb84d764 Jul 24 14:57:38.988636 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:38.988601 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854b698b9d-mjswt" event={"ID":"cb3b5c4a-944f-40bc-8602-7f5832083cd8","Type":"ContainerStarted","Data":"69f7f051a84a508ef2245b0c3f2d602e400e45c93a1dddce15fef29a0c0df473"} Jul 24 14:57:38.989791 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:38.989763 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cn668" event={"ID":"d640e5ab-e9d6-43c6-8081-468e57639679","Type":"ContainerStarted","Data":"1f16405fda3a12e648cbc3e34137fd68bc5033aaa5d9c694148c6165cb84d764"} Jul 24 14:57:39.014570 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:39.014530 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-854b698b9d-mjswt" podStartSLOduration=1.966004453 podStartE2EDuration="5.014515945s" totalImagesPullingTime="3.048511492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:39.013282855 +0000 UTC m=+77.960776467" watchObservedRunningTime="2026-07-24 14:57:39.014515945 +0000 UTC m=+77.962009530" Jul 24 14:57:39.994057 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:39.994023 2557 generic.go:356] "Generic (PLEG): container finished" podID="d640e5ab-e9d6-43c6-8081-468e57639679" containerID="df03581b5e885c35563f351c01fd567332061f0a5e6ea9f5bd2dd84f7d08d84c" exitCode=0 Jul 24 14:57:39.994492 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:39.994113 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cn668" event={"ID":"d640e5ab-e9d6-43c6-8081-468e57639679","Type":"ContainerDied","Data":"df03581b5e885c35563f351c01fd567332061f0a5e6ea9f5bd2dd84f7d08d84c"} Jul 24 14:57:40.348726 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.348632 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b49757f9c-wk6pd"] Jul 24 14:57:40.351938 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.351920 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.354773 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.354751 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 24 14:57:40.354900 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.354797 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 24 14:57:40.354900 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.354751 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mzm76\"" Jul 24 14:57:40.354900 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.354758 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9h4lmceoam5d8\"" Jul 24 14:57:40.354900 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.354844 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 24 14:57:40.355119 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.355047 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 24 14:57:40.355319 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.355307 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 24 14:57:40.364531 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.364511 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b49757f9c-wk6pd"] Jul 24 14:57:40.424208 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.424155 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-tls\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.424416 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.424214 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-grpc-tls\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.424416 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.424289 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.424416 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.424334 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-metrics-client-ca\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.424416 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.424383 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.424416 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.424414 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.424631 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.424466 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.424631 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.424510 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84wv\" (UniqueName: \"kubernetes.io/projected/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-kube-api-access-f84wv\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.525786 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.525752 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.525966 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.525800 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.525966 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.525843 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.525966 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.525870 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84wv\" (UniqueName: \"kubernetes.io/projected/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-kube-api-access-f84wv\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.525966 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.525899 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-tls\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.525966 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.525927 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-grpc-tls\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.525966 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.525957 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.526310 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.525986 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-metrics-client-ca\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.526966 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.526939 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-metrics-client-ca\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.528699 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.528641 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.528818 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.528730 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-tls\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.528927 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.528908 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.529063 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.529042 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.529164 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.529148 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-grpc-tls\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.529473 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.529453 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.534453 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.534428 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84wv\" (UniqueName: \"kubernetes.io/projected/f1ccc53f-2d30-4cda-aab9-9da65207fd8b-kube-api-access-f84wv\") pod \"thanos-querier-6b49757f9c-wk6pd\" (UID: \"f1ccc53f-2d30-4cda-aab9-9da65207fd8b\") " pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.660721 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.660690 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:40.793069 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:40.793036 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b49757f9c-wk6pd"] Jul 24 14:57:40.795959 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:40.795930 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ccc53f_2d30_4cda_aab9_9da65207fd8b.slice/crio-d868b12b3ccf29428bc0e40140ad9fe70f11e3e36f4fef28e20a6d46bbe2c93d WatchSource:0}: Error finding container d868b12b3ccf29428bc0e40140ad9fe70f11e3e36f4fef28e20a6d46bbe2c93d: Status 404 returned error can't find the container with id d868b12b3ccf29428bc0e40140ad9fe70f11e3e36f4fef28e20a6d46bbe2c93d Jul 24 14:57:41.001391 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.001296 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cn668" event={"ID":"d640e5ab-e9d6-43c6-8081-468e57639679","Type":"ContainerStarted","Data":"cb3fb61e46fb6ad8f924ad0540ae033944a5dc119df26ce7aa8eeee3719f9666"} Jul 24 14:57:41.001391 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.001368 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cn668" event={"ID":"d640e5ab-e9d6-43c6-8081-468e57639679","Type":"ContainerStarted","Data":"e154d7493d3d0aef02cffb17301ca9860fa117849489a3106e698dbb4820265b"} Jul 24 14:57:41.002396 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.002377 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" event={"ID":"f1ccc53f-2d30-4cda-aab9-9da65207fd8b","Type":"ContainerStarted","Data":"d868b12b3ccf29428bc0e40140ad9fe70f11e3e36f4fef28e20a6d46bbe2c93d"} Jul 24 14:57:41.024540 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.024492 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cn668" podStartSLOduration=2.979559072 podStartE2EDuration="4.024477106s" totalImagesPullingTime="974.809447ms" totalInitContainerRuntime="70.108587ms" isStatefulPod=true podCreationTimestamp="2026-07-24 14:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:41.023214818 +0000 UTC m=+79.970708407" watchObservedRunningTime="2026-07-24 14:57:41.024477106 +0000 UTC m=+79.971970692" Jul 24 14:57:41.893505 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.893468 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-68cf457585-bqln9"] Jul 24 14:57:41.896806 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.896785 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:41.904732 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.900877 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 14:57:41.904732 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.901008 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8hi2j2ae96ofu\"" Jul 24 14:57:41.904732 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.901604 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 24 14:57:41.904732 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.901925 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tnfpm\"" Jul 24 14:57:41.904732 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.902002 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 24 14:57:41.904732 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.902292 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 24 14:57:41.909165 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:41.909131 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68cf457585-bqln9"] Jul 24 14:57:42.039139 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.039003 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/55497eba-30a8-4b79-a109-bce0b5ca87c8-secret-metrics-server-client-certs\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.039139 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.039049 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55497eba-30a8-4b79-a109-bce0b5ca87c8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.039139 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.039079 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxl6s\" (UniqueName: \"kubernetes.io/projected/55497eba-30a8-4b79-a109-bce0b5ca87c8-kube-api-access-kxl6s\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.039139 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.039108 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/55497eba-30a8-4b79-a109-bce0b5ca87c8-audit-log\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.039671 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.039246 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/55497eba-30a8-4b79-a109-bce0b5ca87c8-metrics-server-audit-profiles\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.039671 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.039295 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/55497eba-30a8-4b79-a109-bce0b5ca87c8-secret-metrics-server-tls\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.039671 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.039323 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/55497eba-30a8-4b79-a109-bce0b5ca87c8-client-ca-bundle\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.140377 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.140336 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/55497eba-30a8-4b79-a109-bce0b5ca87c8-secret-metrics-server-client-certs\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.140572 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.140386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55497eba-30a8-4b79-a109-bce0b5ca87c8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.140572 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.140408 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxl6s\" (UniqueName: \"kubernetes.io/projected/55497eba-30a8-4b79-a109-bce0b5ca87c8-kube-api-access-kxl6s\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.140572 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.140438 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/55497eba-30a8-4b79-a109-bce0b5ca87c8-audit-log\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.140572 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.140531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/55497eba-30a8-4b79-a109-bce0b5ca87c8-metrics-server-audit-profiles\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.140572 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.140561 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/55497eba-30a8-4b79-a109-bce0b5ca87c8-secret-metrics-server-tls\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.140830 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.140579 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/55497eba-30a8-4b79-a109-bce0b5ca87c8-client-ca-bundle\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.141075 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.141049 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/55497eba-30a8-4b79-a109-bce0b5ca87c8-audit-log\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.141306 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.141279 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55497eba-30a8-4b79-a109-bce0b5ca87c8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.141748 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.141720 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/55497eba-30a8-4b79-a109-bce0b5ca87c8-metrics-server-audit-profiles\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.143317 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.143296 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/55497eba-30a8-4b79-a109-bce0b5ca87c8-client-ca-bundle\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.143468 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.143447 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/55497eba-30a8-4b79-a109-bce0b5ca87c8-secret-metrics-server-tls\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.144132 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.144088 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/55497eba-30a8-4b79-a109-bce0b5ca87c8-secret-metrics-server-client-certs\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.150064 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.150034 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxl6s\" (UniqueName: \"kubernetes.io/projected/55497eba-30a8-4b79-a109-bce0b5ca87c8-kube-api-access-kxl6s\") pod \"metrics-server-68cf457585-bqln9\" (UID: \"55497eba-30a8-4b79-a109-bce0b5ca87c8\") " pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.213137 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.213099 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:57:42.793917 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.793888 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-844c66b9bd-sltwr"] Jul 24 14:57:42.798069 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.798051 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:42.800553 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.800533 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 24 14:57:42.800665 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.800533 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 24 14:57:42.801005 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.800983 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 24 14:57:42.801096 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.801032 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 24 14:57:42.801096 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.801075 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bc7wl\"" Jul 24 14:57:42.801320 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.801306 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 24 14:57:42.808215 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.808171 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 24 14:57:42.814461 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.813981 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-844c66b9bd-sltwr"] Jul 24 14:57:42.823232 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.823178 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68cf457585-bqln9"] Jul 24 14:57:42.826878 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:42.826844 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55497eba_30a8_4b79_a109_bce0b5ca87c8.slice/crio-33065a588e5ee4e7e30b8aed77b33d7e7fc5a03ceb7a3d2a271a3e5b7436f2ff WatchSource:0}: Error finding container 33065a588e5ee4e7e30b8aed77b33d7e7fc5a03ceb7a3d2a271a3e5b7436f2ff: Status 404 returned error can't find the container with id 33065a588e5ee4e7e30b8aed77b33d7e7fc5a03ceb7a3d2a271a3e5b7436f2ff Jul 24 14:57:42.948620 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.948542 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc0eb56-9c0f-45a8-b210-3592e268338e-metrics-client-ca\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:42.948620 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.948589 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-telemeter-client-tls\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:42.948620 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.948609 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc0eb56-9c0f-45a8-b210-3592e268338e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:42.948804 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.948628 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc0eb56-9c0f-45a8-b210-3592e268338e-serving-certs-ca-bundle\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:42.948804 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.948644 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-secret-telemeter-client\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:42.948804 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.948727 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-federate-client-tls\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:42.948804 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.948766 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw9nf\" (UniqueName: \"kubernetes.io/projected/ffc0eb56-9c0f-45a8-b210-3592e268338e-kube-api-access-sw9nf\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:42.948919 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:42.948823 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.012940 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.012906 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" event={"ID":"f1ccc53f-2d30-4cda-aab9-9da65207fd8b","Type":"ContainerStarted","Data":"f162cdb07ff0abb6cfff9daf1993af8cedd82289f933aa254ddd22029907e564"} Jul 24 14:57:43.013131 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.012978 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" event={"ID":"f1ccc53f-2d30-4cda-aab9-9da65207fd8b","Type":"ContainerStarted","Data":"1bca52ea3ca4fc9c01f5be017042e36d4e6d13ac4fd643d21ddb6530cbd82edb"} Jul 24 14:57:43.013131 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.013015 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" event={"ID":"f1ccc53f-2d30-4cda-aab9-9da65207fd8b","Type":"ContainerStarted","Data":"6a44c1a69862cc938bec0ccb060f1134d58c5738853e02ff141b17deb04aa40c"} Jul 24 14:57:43.013961 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.013938 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68cf457585-bqln9" event={"ID":"55497eba-30a8-4b79-a109-bce0b5ca87c8","Type":"ContainerStarted","Data":"33065a588e5ee4e7e30b8aed77b33d7e7fc5a03ceb7a3d2a271a3e5b7436f2ff"} Jul 24 14:57:43.049418 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.049387 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.049699 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.049422 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc0eb56-9c0f-45a8-b210-3592e268338e-metrics-client-ca\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.049699 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.049593 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-telemeter-client-tls\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.049699 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.049627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc0eb56-9c0f-45a8-b210-3592e268338e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.049699 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.049663 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc0eb56-9c0f-45a8-b210-3592e268338e-serving-certs-ca-bundle\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.049699 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.049685 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-secret-telemeter-client\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.049947 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.049733 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-federate-client-tls\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.049947 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.049759 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw9nf\" (UniqueName: \"kubernetes.io/projected/ffc0eb56-9c0f-45a8-b210-3592e268338e-kube-api-access-sw9nf\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.050267 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.050243 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc0eb56-9c0f-45a8-b210-3592e268338e-metrics-client-ca\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.050564 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.050479 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc0eb56-9c0f-45a8-b210-3592e268338e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.050696 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.050658 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc0eb56-9c0f-45a8-b210-3592e268338e-serving-certs-ca-bundle\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.052408 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.052385 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-federate-client-tls\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.052512 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.052423 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-secret-telemeter-client\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.052512 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.052488 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.052628 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.052616 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ffc0eb56-9c0f-45a8-b210-3592e268338e-telemeter-client-tls\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.061582 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.061562 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw9nf\" (UniqueName: \"kubernetes.io/projected/ffc0eb56-9c0f-45a8-b210-3592e268338e-kube-api-access-sw9nf\") pod \"telemeter-client-844c66b9bd-sltwr\" (UID: \"ffc0eb56-9c0f-45a8-b210-3592e268338e\") " pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.109627 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.109598 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" Jul 24 14:57:43.243144 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:43.243117 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-844c66b9bd-sltwr"] Jul 24 14:57:43.245544 ip-10-0-139-100 kubenswrapper[2557]: W0724 14:57:43.245517 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc0eb56_9c0f_45a8_b210_3592e268338e.slice/crio-47bab9aa8800a206b62b26c9c795c730f9cffe921d539af7f3130286970db05f WatchSource:0}: Error finding container 47bab9aa8800a206b62b26c9c795c730f9cffe921d539af7f3130286970db05f: Status 404 returned error can't find the container with id 47bab9aa8800a206b62b26c9c795c730f9cffe921d539af7f3130286970db05f Jul 24 14:57:44.024606 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:44.024570 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" event={"ID":"f1ccc53f-2d30-4cda-aab9-9da65207fd8b","Type":"ContainerStarted","Data":"667f34078604f33560c03fe3f3ca4180dfbcc359d817091e7a04d09ff6724600"} Jul 24 14:57:44.024748 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:44.024661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" event={"ID":"f1ccc53f-2d30-4cda-aab9-9da65207fd8b","Type":"ContainerStarted","Data":"4f5b847fd61330fccccc1ec12cfe20e75ebb82e9ed5024c2928c8636df24b450"} Jul 24 14:57:44.027054 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:44.027027 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" event={"ID":"ffc0eb56-9c0f-45a8-b210-3592e268338e","Type":"ContainerStarted","Data":"47bab9aa8800a206b62b26c9c795c730f9cffe921d539af7f3130286970db05f"} Jul 24 14:57:45.034363 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:45.034322 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" event={"ID":"f1ccc53f-2d30-4cda-aab9-9da65207fd8b","Type":"ContainerStarted","Data":"9b0970f34763adc34b84dec47a0ba1bb95a20f5b1edbc1f747398911347cb544"} Jul 24 14:57:45.034657 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:45.034472 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:45.062693 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:45.062650 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" podStartSLOduration=2.218948335 podStartE2EDuration="5.062634443s" totalImagesPullingTime="2.843686108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:57:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:45.060897334 +0000 UTC m=+84.008390919" watchObservedRunningTime="2026-07-24 14:57:45.062634443 +0000 UTC m=+84.010128026" Jul 24 14:57:45.129378 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:45.129344 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:45.129533 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:45.129438 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:45.133838 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:45.133807 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:46.039666 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:46.039630 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" event={"ID":"ffc0eb56-9c0f-45a8-b210-3592e268338e","Type":"ContainerStarted","Data":"054ddab09270aa766ffb330d68298f3554dd781d98f54a90b5c58ba0dedb2c31"} Jul 24 14:57:46.041200 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:46.041164 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68cf457585-bqln9" event={"ID":"55497eba-30a8-4b79-a109-bce0b5ca87c8","Type":"ContainerStarted","Data":"27d2fe7b95eacba826788e889e3dc6354fcffb2a7be551f06ddaeff22a1207d7"} Jul 24 14:57:46.045033 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:46.045012 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:57:46.075872 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:46.075810 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-68cf457585-bqln9" podStartSLOduration=2.888570693 podStartE2EDuration="5.075790628s" totalImagesPullingTime="2.187219935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:46.07471294 +0000 UTC m=+85.022206526" watchObservedRunningTime="2026-07-24 14:57:46.075790628 +0000 UTC m=+85.023284215" Jul 24 14:57:47.046939 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:47.046892 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" event={"ID":"ffc0eb56-9c0f-45a8-b210-3592e268338e","Type":"ContainerStarted","Data":"2739b21118339dbe987786d34a989a3364d9e890d520f4b6205b4f2ac46aea78"} Jul 24 14:57:47.047367 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:47.046982 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" event={"ID":"ffc0eb56-9c0f-45a8-b210-3592e268338e","Type":"ContainerStarted","Data":"9b528eddebaa1407395d7cf4420e3ceb2cdb09f5f72c147c683c7311c70860c4"} Jul 24 14:57:47.073140 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:47.073083 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-844c66b9bd-sltwr" podStartSLOduration=2.349456049 podStartE2EDuration="5.073064474s" totalImagesPullingTime="2.723608425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:57:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:57:47.072319019 +0000 UTC m=+86.019812636" watchObservedRunningTime="2026-07-24 14:57:47.073064474 +0000 UTC m=+86.020558061" Jul 24 14:57:51.047245 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:51.047218 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b49757f9c-wk6pd" Jul 24 14:57:58.984558 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:57:58.984526 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854b698b9d-mjswt"] Jul 24 14:58:02.213724 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:02.213686 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:58:02.214110 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:02.213752 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:58:05.978956 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:05.978898 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2hd8j" Jul 24 14:58:22.218502 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:22.218472 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:58:22.222174 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:22.222150 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-68cf457585-bqln9" Jul 24 14:58:24.003743 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.003674 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-854b698b9d-mjswt" podUID="cb3b5c4a-944f-40bc-8602-7f5832083cd8" containerName="console" containerID="cri-o://69f7f051a84a508ef2245b0c3f2d602e400e45c93a1dddce15fef29a0c0df473" gracePeriod=15 Jul 24 14:58:24.181951 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.181919 2557 generic.go:356] "Generic (PLEG): container finished" podID="cb3b5c4a-944f-40bc-8602-7f5832083cd8" containerID="69f7f051a84a508ef2245b0c3f2d602e400e45c93a1dddce15fef29a0c0df473" exitCode=0 Jul 24 14:58:24.182077 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.181988 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854b698b9d-mjswt" event={"ID":"cb3b5c4a-944f-40bc-8602-7f5832083cd8","Type":"ContainerDied","Data":"69f7f051a84a508ef2245b0c3f2d602e400e45c93a1dddce15fef29a0c0df473"} Jul 24 14:58:24.248037 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.248013 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:58:24.260169 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.260081 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-oauth-serving-cert\") pod \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " Jul 24 14:58:24.260169 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.260118 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzxv8\" (UniqueName: \"kubernetes.io/projected/cb3b5c4a-944f-40bc-8602-7f5832083cd8-kube-api-access-rzxv8\") pod \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " Jul 24 14:58:24.260169 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.260141 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-serving-cert\") pod \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " Jul 24 14:58:24.260169 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.260159 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-config\") pod \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " Jul 24 14:58:24.260455 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.260218 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-service-ca\") pod \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " Jul 24 14:58:24.260455 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.260250 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-oauth-config\") pod \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\" (UID: \"cb3b5c4a-944f-40bc-8602-7f5832083cd8\") " Jul 24 14:58:24.260602 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.260562 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cb3b5c4a-944f-40bc-8602-7f5832083cd8" (UID: "cb3b5c4a-944f-40bc-8602-7f5832083cd8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:58:24.260675 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.260629 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-config" (OuterVolumeSpecName: "console-config") pod "cb3b5c4a-944f-40bc-8602-7f5832083cd8" (UID: "cb3b5c4a-944f-40bc-8602-7f5832083cd8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:58:24.260675 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.260632 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-service-ca" (OuterVolumeSpecName: "service-ca") pod "cb3b5c4a-944f-40bc-8602-7f5832083cd8" (UID: "cb3b5c4a-944f-40bc-8602-7f5832083cd8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:58:24.262479 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.262450 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cb3b5c4a-944f-40bc-8602-7f5832083cd8" (UID: "cb3b5c4a-944f-40bc-8602-7f5832083cd8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:58:24.262572 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.262480 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cb3b5c4a-944f-40bc-8602-7f5832083cd8" (UID: "cb3b5c4a-944f-40bc-8602-7f5832083cd8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:58:24.262572 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.262563 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3b5c4a-944f-40bc-8602-7f5832083cd8-kube-api-access-rzxv8" (OuterVolumeSpecName: "kube-api-access-rzxv8") pod "cb3b5c4a-944f-40bc-8602-7f5832083cd8" (UID: "cb3b5c4a-944f-40bc-8602-7f5832083cd8"). InnerVolumeSpecName "kube-api-access-rzxv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:58:24.361407 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.361377 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 14:58:24.361407 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.361403 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-oauth-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 14:58:24.361407 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.361413 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-oauth-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 14:58:24.361632 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.361422 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzxv8\" (UniqueName: \"kubernetes.io/projected/cb3b5c4a-944f-40bc-8602-7f5832083cd8-kube-api-access-rzxv8\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 14:58:24.361632 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.361431 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 14:58:24.361632 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:24.361439 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb3b5c4a-944f-40bc-8602-7f5832083cd8-console-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 14:58:25.186847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:25.186809 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854b698b9d-mjswt" event={"ID":"cb3b5c4a-944f-40bc-8602-7f5832083cd8","Type":"ContainerDied","Data":"10c056ed3de9874795c7d54a85dcd5d817030595063de58e8fa1d2bde2993f98"} Jul 24 14:58:25.186847 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:25.186826 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854b698b9d-mjswt" Jul 24 14:58:25.187373 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:25.186862 2557 scope.go:117] "RemoveContainer" containerID="69f7f051a84a508ef2245b0c3f2d602e400e45c93a1dddce15fef29a0c0df473" Jul 24 14:58:25.209787 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:25.209760 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854b698b9d-mjswt"] Jul 24 14:58:25.216731 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:25.216712 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-854b698b9d-mjswt"] Jul 24 14:58:25.586567 ip-10-0-139-100 kubenswrapper[2557]: I0724 14:58:25.586487 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb3b5c4a-944f-40bc-8602-7f5832083cd8" path="/var/lib/kubelet/pods/cb3b5c4a-944f-40bc-8602-7f5832083cd8/volumes" Jul 24 15:00:53.807461 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.807377 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz"] Jul 24 15:00:53.807904 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.807636 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3b5c4a-944f-40bc-8602-7f5832083cd8" containerName="console" Jul 24 15:00:53.807904 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.807647 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3b5c4a-944f-40bc-8602-7f5832083cd8" containerName="console" Jul 24 15:00:53.807904 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.807692 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3b5c4a-944f-40bc-8602-7f5832083cd8" containerName="console" Jul 24 15:00:53.810434 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.810419 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" Jul 24 15:00:53.813295 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.813275 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-w8x4f\"" Jul 24 15:00:53.813438 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.813314 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 15:00:53.814243 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.814224 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 15:00:53.821667 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.821643 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz"] Jul 24 15:00:53.959360 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.959317 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgh4\" (UniqueName: \"kubernetes.io/projected/5b2ed007-4982-4b87-9262-61fbb5633e4a-kube-api-access-gtgh4\") pod \"cert-manager-cainjector-559cdd8c87-9hzmz\" (UID: \"5b2ed007-4982-4b87-9262-61fbb5633e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" Jul 24 15:00:53.959535 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:53.959442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b2ed007-4982-4b87-9262-61fbb5633e4a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9hzmz\" (UID: \"5b2ed007-4982-4b87-9262-61fbb5633e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" Jul 24 15:00:54.060671 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:54.060568 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b2ed007-4982-4b87-9262-61fbb5633e4a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9hzmz\" (UID: \"5b2ed007-4982-4b87-9262-61fbb5633e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" Jul 24 15:00:54.060671 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:54.060621 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgh4\" (UniqueName: \"kubernetes.io/projected/5b2ed007-4982-4b87-9262-61fbb5633e4a-kube-api-access-gtgh4\") pod \"cert-manager-cainjector-559cdd8c87-9hzmz\" (UID: \"5b2ed007-4982-4b87-9262-61fbb5633e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" Jul 24 15:00:54.070965 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:54.070931 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b2ed007-4982-4b87-9262-61fbb5633e4a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9hzmz\" (UID: \"5b2ed007-4982-4b87-9262-61fbb5633e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" Jul 24 15:00:54.071090 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:54.071036 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgh4\" (UniqueName: \"kubernetes.io/projected/5b2ed007-4982-4b87-9262-61fbb5633e4a-kube-api-access-gtgh4\") pod \"cert-manager-cainjector-559cdd8c87-9hzmz\" (UID: \"5b2ed007-4982-4b87-9262-61fbb5633e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" Jul 24 15:00:54.119613 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:54.119566 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" Jul 24 15:00:54.249821 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:54.249793 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz"] Jul 24 15:00:54.252267 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:00:54.252235 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b2ed007_4982_4b87_9262_61fbb5633e4a.slice/crio-77b742c642bd758e8bee57f98b5d766f047e030948d756d564b83348707b877a WatchSource:0}: Error finding container 77b742c642bd758e8bee57f98b5d766f047e030948d756d564b83348707b877a: Status 404 returned error can't find the container with id 77b742c642bd758e8bee57f98b5d766f047e030948d756d564b83348707b877a Jul 24 15:00:54.699300 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:54.699265 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" event={"ID":"5b2ed007-4982-4b87-9262-61fbb5633e4a","Type":"ContainerStarted","Data":"77b742c642bd758e8bee57f98b5d766f047e030948d756d564b83348707b877a"} Jul 24 15:00:57.714434 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:57.714393 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" event={"ID":"5b2ed007-4982-4b87-9262-61fbb5633e4a","Type":"ContainerStarted","Data":"08417a0df76b987d2c6690c63503ef6cbecdcf893cd9a9ac41b62414cbac81b2"} Jul 24 15:00:57.734069 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:00:57.733937 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9hzmz" podStartSLOduration=1.505446871 podStartE2EDuration="4.733914177s" totalImagesPullingTime="3.228467306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:00:57.733770595 +0000 UTC m=+276.681264183" watchObservedRunningTime="2026-07-24 15:00:57.733914177 +0000 UTC m=+276.681407763" Jul 24 15:01:21.404431 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:21.404399 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 15:01:23.409733 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.409701 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf"] Jul 24 15:01:23.411633 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.411616 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.414635 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.414609 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 15:01:23.414756 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.414615 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 15:01:23.414756 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.414648 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 15:01:23.414938 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.414923 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lmn65\"" Jul 24 15:01:23.415389 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.415375 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 15:01:23.428609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.428574 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf"] Jul 24 15:01:23.472599 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.472567 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe20b512-4076-4eea-b66b-e2b4216df9c1-manifests\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.472599 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.472606 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe20b512-4076-4eea-b66b-e2b4216df9c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.472862 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.472624 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe20b512-4076-4eea-b66b-e2b4216df9c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.472862 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.472657 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe20b512-4076-4eea-b66b-e2b4216df9c1-tmp\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.472862 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.472717 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tfvh\" (UniqueName: \"kubernetes.io/projected/fe20b512-4076-4eea-b66b-e2b4216df9c1-kube-api-access-7tfvh\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.573965 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.573922 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe20b512-4076-4eea-b66b-e2b4216df9c1-manifests\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.573965 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.573967 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe20b512-4076-4eea-b66b-e2b4216df9c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.574238 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.573984 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe20b512-4076-4eea-b66b-e2b4216df9c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.574238 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.574007 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe20b512-4076-4eea-b66b-e2b4216df9c1-tmp\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.574238 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.574027 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tfvh\" (UniqueName: \"kubernetes.io/projected/fe20b512-4076-4eea-b66b-e2b4216df9c1-kube-api-access-7tfvh\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.574442 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.574419 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe20b512-4076-4eea-b66b-e2b4216df9c1-manifests\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.576403 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.576376 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe20b512-4076-4eea-b66b-e2b4216df9c1-tmp\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.576606 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.576588 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe20b512-4076-4eea-b66b-e2b4216df9c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.576737 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.576715 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe20b512-4076-4eea-b66b-e2b4216df9c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.587501 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.587472 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tfvh\" (UniqueName: \"kubernetes.io/projected/fe20b512-4076-4eea-b66b-e2b4216df9c1-kube-api-access-7tfvh\") pod \"opendatahub-operator-controller-manager-7f5584866b-9b7vf\" (UID: \"fe20b512-4076-4eea-b66b-e2b4216df9c1\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.721829 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.721738 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:23.859480 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.859456 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf"] Jul 24 15:01:23.861944 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:01:23.861923 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe20b512_4076_4eea_b66b_e2b4216df9c1.slice/crio-aef197a03429fb816fb40d74eb2ea808f507e8e009017c3987f9d1965b59f9ea WatchSource:0}: Error finding container aef197a03429fb816fb40d74eb2ea808f507e8e009017c3987f9d1965b59f9ea: Status 404 returned error can't find the container with id aef197a03429fb816fb40d74eb2ea808f507e8e009017c3987f9d1965b59f9ea Jul 24 15:01:23.863840 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:23.863819 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:01:24.812077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:24.812016 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" event={"ID":"fe20b512-4076-4eea-b66b-e2b4216df9c1","Type":"ContainerStarted","Data":"aef197a03429fb816fb40d74eb2ea808f507e8e009017c3987f9d1965b59f9ea"} Jul 24 15:01:27.826234 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:27.826130 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" event={"ID":"fe20b512-4076-4eea-b66b-e2b4216df9c1","Type":"ContainerStarted","Data":"f0e0c0cef60e244b24d9a6f594430c16d1abe17841d045b50bc649e521a10d5f"} Jul 24 15:01:28.830804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:28.830768 2557 generic.go:356] "Generic (PLEG): container finished" podID="fe20b512-4076-4eea-b66b-e2b4216df9c1" containerID="f0e0c0cef60e244b24d9a6f594430c16d1abe17841d045b50bc649e521a10d5f" exitCode=0 Jul 24 15:01:28.831314 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:28.830827 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" event={"ID":"fe20b512-4076-4eea-b66b-e2b4216df9c1","Type":"ContainerDied","Data":"f0e0c0cef60e244b24d9a6f594430c16d1abe17841d045b50bc649e521a10d5f"} Jul 24 15:01:29.836488 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:29.836452 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" event={"ID":"fe20b512-4076-4eea-b66b-e2b4216df9c1","Type":"ContainerStarted","Data":"fd764792639fa898ea512e39a82ea0e42c32b42ef8393ee262b6bf67569b78f7"} Jul 24 15:01:29.836851 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:29.836586 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:29.867047 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:29.866998 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" podStartSLOduration=2.460039042 podStartE2EDuration="6.866981819s" totalImagesPullingTime="3.327171703s" totalInitContainerRuntime="1.079771074s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:01:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:01:29.866915483 +0000 UTC m=+308.814409083" watchObservedRunningTime="2026-07-24 15:01:29.866981819 +0000 UTC m=+308.814475405" Jul 24 15:01:40.842547 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:40.842519 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5584866b-9b7vf" Jul 24 15:01:43.469098 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.469060 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cx55j"] Jul 24 15:01:43.472162 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.472146 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:01:43.475085 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.475062 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-z7s7c\"" Jul 24 15:01:43.475221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.475139 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 15:01:43.495541 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.495511 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cx55j"] Jul 24 15:01:43.534323 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.534287 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3e11020-e4db-4714-99e4-3519a38e2ac5-cert\") pod \"odh-model-controller-7998c67dc4-cx55j\" (UID: \"a3e11020-e4db-4714-99e4-3519a38e2ac5\") " pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:01:43.534499 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.534342 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r27c\" (UniqueName: \"kubernetes.io/projected/a3e11020-e4db-4714-99e4-3519a38e2ac5-kube-api-access-6r27c\") pod \"odh-model-controller-7998c67dc4-cx55j\" (UID: \"a3e11020-e4db-4714-99e4-3519a38e2ac5\") " pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:01:43.634759 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.634725 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3e11020-e4db-4714-99e4-3519a38e2ac5-cert\") pod \"odh-model-controller-7998c67dc4-cx55j\" (UID: \"a3e11020-e4db-4714-99e4-3519a38e2ac5\") " pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:01:43.634945 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.634776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r27c\" (UniqueName: \"kubernetes.io/projected/a3e11020-e4db-4714-99e4-3519a38e2ac5-kube-api-access-6r27c\") pod \"odh-model-controller-7998c67dc4-cx55j\" (UID: \"a3e11020-e4db-4714-99e4-3519a38e2ac5\") " pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:01:43.637359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.637333 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3e11020-e4db-4714-99e4-3519a38e2ac5-cert\") pod \"odh-model-controller-7998c67dc4-cx55j\" (UID: \"a3e11020-e4db-4714-99e4-3519a38e2ac5\") " pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:01:43.652261 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.652232 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r27c\" (UniqueName: \"kubernetes.io/projected/a3e11020-e4db-4714-99e4-3519a38e2ac5-kube-api-access-6r27c\") pod \"odh-model-controller-7998c67dc4-cx55j\" (UID: \"a3e11020-e4db-4714-99e4-3519a38e2ac5\") " pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:01:43.781986 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.781900 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:01:43.916521 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:43.916491 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cx55j"] Jul 24 15:01:43.919698 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:01:43.919673 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e11020_e4db_4714_99e4_3519a38e2ac5.slice/crio-a6b07d4e67d620676376042326795ed50de87edbcc7bc1c9a5bbdc490949e1c6 WatchSource:0}: Error finding container a6b07d4e67d620676376042326795ed50de87edbcc7bc1c9a5bbdc490949e1c6: Status 404 returned error can't find the container with id a6b07d4e67d620676376042326795ed50de87edbcc7bc1c9a5bbdc490949e1c6 Jul 24 15:01:44.169780 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.169748 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn"] Jul 24 15:01:44.174077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.174061 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.177892 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.177860 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 15:01:44.178045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.177860 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 15:01:44.178045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.177923 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jnrv5\"" Jul 24 15:01:44.178045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.177927 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:01:44.178045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.177867 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 15:01:44.178045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.177925 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 15:01:44.183593 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.183570 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn"] Jul 24 15:01:44.240007 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.239972 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6svst\" (UniqueName: \"kubernetes.io/projected/b4b953b5-1229-4969-b243-226a97234d6a-kube-api-access-6svst\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.240007 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.240011 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4b953b5-1229-4969-b243-226a97234d6a-cert\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.240244 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.240032 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b4b953b5-1229-4969-b243-226a97234d6a-manager-config\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.240244 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.240076 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4b953b5-1229-4969-b243-226a97234d6a-metrics-cert\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.340713 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.340683 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6svst\" (UniqueName: \"kubernetes.io/projected/b4b953b5-1229-4969-b243-226a97234d6a-kube-api-access-6svst\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.340900 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.340731 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4b953b5-1229-4969-b243-226a97234d6a-cert\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.340900 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.340764 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b4b953b5-1229-4969-b243-226a97234d6a-manager-config\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.340900 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.340814 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4b953b5-1229-4969-b243-226a97234d6a-metrics-cert\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.341570 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.341468 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b4b953b5-1229-4969-b243-226a97234d6a-manager-config\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.343224 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.343202 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4b953b5-1229-4969-b243-226a97234d6a-cert\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.343319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.343266 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4b953b5-1229-4969-b243-226a97234d6a-metrics-cert\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.367101 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.367071 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6svst\" (UniqueName: \"kubernetes.io/projected/b4b953b5-1229-4969-b243-226a97234d6a-kube-api-access-6svst\") pod \"lws-controller-manager-54d8b9c4f-lnkkn\" (UID: \"b4b953b5-1229-4969-b243-226a97234d6a\") " pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.483571 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.483485 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:44.624483 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.624415 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn"] Jul 24 15:01:44.627792 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:01:44.627761 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b953b5_1229_4969_b243_226a97234d6a.slice/crio-512b425a7de23f579326fd2af8ccb3f2cc30fc4c4096f3f20c9c34eba7f08620 WatchSource:0}: Error finding container 512b425a7de23f579326fd2af8ccb3f2cc30fc4c4096f3f20c9c34eba7f08620: Status 404 returned error can't find the container with id 512b425a7de23f579326fd2af8ccb3f2cc30fc4c4096f3f20c9c34eba7f08620 Jul 24 15:01:44.894516 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.894430 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" event={"ID":"b4b953b5-1229-4969-b243-226a97234d6a","Type":"ContainerStarted","Data":"512b425a7de23f579326fd2af8ccb3f2cc30fc4c4096f3f20c9c34eba7f08620"} Jul 24 15:01:44.895936 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:44.895901 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" event={"ID":"a3e11020-e4db-4714-99e4-3519a38e2ac5","Type":"ContainerStarted","Data":"a6b07d4e67d620676376042326795ed50de87edbcc7bc1c9a5bbdc490949e1c6"} Jul 24 15:01:48.913543 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:48.913509 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" event={"ID":"b4b953b5-1229-4969-b243-226a97234d6a","Type":"ContainerStarted","Data":"8a492fb174d64b2009506debea8091674813da230ece53d9dc34eb786f8c8e64"} Jul 24 15:01:48.914013 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:48.913593 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:48.914906 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:48.914882 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" event={"ID":"a3e11020-e4db-4714-99e4-3519a38e2ac5","Type":"ContainerStarted","Data":"31d5c2ccce1092f92530fe9b32c61ce72408808727d32a3e1f576d4fdf64372c"} Jul 24 15:01:48.915021 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:48.915009 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:01:48.932003 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:48.931957 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" podStartSLOduration=1.636575035 podStartE2EDuration="4.931942341s" totalImagesPullingTime="3.295367306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:01:48.930048483 +0000 UTC m=+327.877542070" watchObservedRunningTime="2026-07-24 15:01:48.931942341 +0000 UTC m=+327.879435928" Jul 24 15:01:48.948037 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:48.947989 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" podStartSLOduration=1.990663306 podStartE2EDuration="5.947975341s" totalImagesPullingTime="3.957312035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:01:48.947464674 +0000 UTC m=+327.894958260" watchObservedRunningTime="2026-07-24 15:01:48.947975341 +0000 UTC m=+327.895468927" Jul 24 15:01:51.639051 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.639012 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd"] Jul 24 15:01:51.642763 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.642740 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.646286 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.646269 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 15:01:51.646553 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.646537 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 15:01:51.646822 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.646808 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 15:01:51.646869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.646807 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-m2ldv\"" Jul 24 15:01:51.657282 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.657259 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd"] Jul 24 15:01:51.806395 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.806362 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jpr8\" (UniqueName: \"kubernetes.io/projected/f7482f19-8e2f-42c4-81cf-cc12517a6c18-kube-api-access-7jpr8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.806395 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.806398 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.806625 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.806423 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.806625 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.806503 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.806625 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.806545 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.806625 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.806563 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.806625 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.806607 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.806801 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.806636 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.806801 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.806676 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907038 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907011 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907060 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jpr8\" (UniqueName: \"kubernetes.io/projected/f7482f19-8e2f-42c4-81cf-cc12517a6c18-kube-api-access-7jpr8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907082 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907112 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907142 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907169 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907520 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907341 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907520 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907420 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907645 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907621 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907699 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907672 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907759 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907705 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.907820 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907783 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.908005 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.907984 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.909593 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.909573 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.909685 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.909628 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.916565 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.916542 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7482f19-8e2f-42c4-81cf-cc12517a6c18-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.917178 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.917157 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jpr8\" (UniqueName: \"kubernetes.io/projected/f7482f19-8e2f-42c4-81cf-cc12517a6c18-kube-api-access-7jpr8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd\" (UID: \"f7482f19-8e2f-42c4-81cf-cc12517a6c18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.952777 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.952745 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:51.981418 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.981393 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n"] Jul 24 15:01:51.984909 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.984891 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:51.997556 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.997530 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mgbrv\"" Jul 24 15:01:51.998650 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.998630 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 15:01:51.998755 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:51.998703 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 15:01:52.001431 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.001413 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 15:01:52.023045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.022953 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n"] Jul 24 15:01:52.109413 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.109383 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.109413 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.109420 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwp5\" (UniqueName: \"kubernetes.io/projected/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-kube-api-access-2mwp5\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.109617 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.109451 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.109617 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.109493 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.109617 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.109598 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.109709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.109640 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.109709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.109662 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.122816 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.122792 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd"] Jul 24 15:01:52.125032 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:01:52.124988 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7482f19_8e2f_42c4_81cf_cc12517a6c18.slice/crio-e2b6ddd7a1647b41c8a80a060beb8c5f230a45d21542b3b0ef7ccc9b0fbe570c WatchSource:0}: Error finding container e2b6ddd7a1647b41c8a80a060beb8c5f230a45d21542b3b0ef7ccc9b0fbe570c: Status 404 returned error can't find the container with id e2b6ddd7a1647b41c8a80a060beb8c5f230a45d21542b3b0ef7ccc9b0fbe570c Jul 24 15:01:52.210065 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.209982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.210065 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.210017 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwp5\" (UniqueName: \"kubernetes.io/projected/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-kube-api-access-2mwp5\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.210065 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.210035 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.210065 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.210057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.210405 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.210138 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.210405 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.210177 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.210405 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.210383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.210955 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.210905 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.212762 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.212739 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.212911 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.212891 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.213018 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.213003 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.213087 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.213036 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.220712 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.220691 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwp5\" (UniqueName: \"kubernetes.io/projected/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-kube-api-access-2mwp5\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.223387 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.223365 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4829a10c-fab4-4195-8d5f-0cdbd8b90ccb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6t92n\" (UID: \"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.294874 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.294838 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:52.436097 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.436075 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n"] Jul 24 15:01:52.437943 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:01:52.437924 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4829a10c_fab4_4195_8d5f_0cdbd8b90ccb.slice/crio-2ce18a7f30089a07a7ef0684545f6ed65b46e1d3627c91b91ec3338262a0dbf3 WatchSource:0}: Error finding container 2ce18a7f30089a07a7ef0684545f6ed65b46e1d3627c91b91ec3338262a0dbf3: Status 404 returned error can't find the container with id 2ce18a7f30089a07a7ef0684545f6ed65b46e1d3627c91b91ec3338262a0dbf3 Jul 24 15:01:52.931237 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.931178 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" event={"ID":"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb","Type":"ContainerStarted","Data":"2ce18a7f30089a07a7ef0684545f6ed65b46e1d3627c91b91ec3338262a0dbf3"} Jul 24 15:01:52.932333 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:52.932306 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" event={"ID":"f7482f19-8e2f-42c4-81cf-cc12517a6c18","Type":"ContainerStarted","Data":"e2b6ddd7a1647b41c8a80a060beb8c5f230a45d21542b3b0ef7ccc9b0fbe570c"} Jul 24 15:01:56.182002 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:56.181958 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:01:56.182401 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:56.182030 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:01:56.185709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:56.185678 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:01:56.185795 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:56.185770 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:01:56.185833 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:56.185798 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:01:56.958557 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:56.958512 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" event={"ID":"4829a10c-fab4-4195-8d5f-0cdbd8b90ccb","Type":"ContainerStarted","Data":"9e881b7182d9c00b917dc90de177e943c8c9841b0c0e52c3f94173940f691c21"} Jul 24 15:01:56.958759 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:56.958609 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:56.959974 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:56.959952 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" event={"ID":"f7482f19-8e2f-42c4-81cf-cc12517a6c18","Type":"ContainerStarted","Data":"ff8ea582d52fd8f58c325f6a175ac2c5bff0c60f4c566adce5097658b0bc756a"} Jul 24 15:01:56.996323 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:56.996276 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" podStartSLOduration=2.254440093 podStartE2EDuration="5.996260807s" totalImagesPullingTime="3.741820714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:01:56.994305775 +0000 UTC m=+335.941799385" watchObservedRunningTime="2026-07-24 15:01:56.996260807 +0000 UTC m=+335.943754394" Jul 24 15:01:57.052057 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:57.052001 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" podStartSLOduration=1.993528685 podStartE2EDuration="6.051980204s" totalImagesPullingTime="4.058451519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:01:57.050677673 +0000 UTC m=+335.998171279" watchObservedRunningTime="2026-07-24 15:01:57.051980204 +0000 UTC m=+335.999473789" Jul 24 15:01:57.953667 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:57.953632 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:57.958214 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:57.958165 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:57.963884 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:57.963856 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:57.964575 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:57.964553 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd" Jul 24 15:01:57.965039 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:57.965020 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6t92n" Jul 24 15:01:59.921552 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:59.921523 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54d8b9c4f-lnkkn" Jul 24 15:01:59.921952 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:01:59.921600 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-cx55j" Jul 24 15:04:52.925953 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:52.925918 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4"] Jul 24 15:04:52.929090 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:52.929072 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:04:52.931802 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:52.931772 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:04:52.931942 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:52.931839 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 15:04:52.931942 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:52.931848 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8zrcz\"" Jul 24 15:04:52.932934 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:52.932881 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 15:04:52.938985 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:52.938961 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4"] Jul 24 15:04:53.014357 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:53.014321 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kdvw\" (UniqueName: \"kubernetes.io/projected/8c989cb2-ce6e-4ea6-baf6-482f737eadb3-kube-api-access-8kdvw\") pod \"postgres-69fb4594cf-kpqp4\" (UID: \"8c989cb2-ce6e-4ea6-baf6-482f737eadb3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:04:53.014550 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:53.014392 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8c989cb2-ce6e-4ea6-baf6-482f737eadb3-data\") pod \"postgres-69fb4594cf-kpqp4\" (UID: \"8c989cb2-ce6e-4ea6-baf6-482f737eadb3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:04:53.115488 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:53.115452 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kdvw\" (UniqueName: \"kubernetes.io/projected/8c989cb2-ce6e-4ea6-baf6-482f737eadb3-kube-api-access-8kdvw\") pod \"postgres-69fb4594cf-kpqp4\" (UID: \"8c989cb2-ce6e-4ea6-baf6-482f737eadb3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:04:53.115670 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:53.115530 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8c989cb2-ce6e-4ea6-baf6-482f737eadb3-data\") pod \"postgres-69fb4594cf-kpqp4\" (UID: \"8c989cb2-ce6e-4ea6-baf6-482f737eadb3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:04:53.115911 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:53.115895 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8c989cb2-ce6e-4ea6-baf6-482f737eadb3-data\") pod \"postgres-69fb4594cf-kpqp4\" (UID: \"8c989cb2-ce6e-4ea6-baf6-482f737eadb3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:04:53.123874 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:53.123847 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kdvw\" (UniqueName: \"kubernetes.io/projected/8c989cb2-ce6e-4ea6-baf6-482f737eadb3-kube-api-access-8kdvw\") pod \"postgres-69fb4594cf-kpqp4\" (UID: \"8c989cb2-ce6e-4ea6-baf6-482f737eadb3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:04:53.240799 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:53.240706 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:04:53.375075 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:53.375035 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4"] Jul 24 15:04:53.379155 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:04:53.379122 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c989cb2_ce6e_4ea6_baf6_482f737eadb3.slice/crio-abc4bd8b134cfa8692e9efde3d1fae22ce998c3513e30be1828c429e07e6017b WatchSource:0}: Error finding container abc4bd8b134cfa8692e9efde3d1fae22ce998c3513e30be1828c429e07e6017b: Status 404 returned error can't find the container with id abc4bd8b134cfa8692e9efde3d1fae22ce998c3513e30be1828c429e07e6017b Jul 24 15:04:53.671150 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:04:53.671115 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" event={"ID":"8c989cb2-ce6e-4ea6-baf6-482f737eadb3","Type":"ContainerStarted","Data":"abc4bd8b134cfa8692e9efde3d1fae22ce998c3513e30be1828c429e07e6017b"} Jul 24 15:05:00.702409 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:05:00.702364 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" event={"ID":"8c989cb2-ce6e-4ea6-baf6-482f737eadb3","Type":"ContainerStarted","Data":"acd6c6bfa3cb4f57577da603fd3272ce1b6d330264782630a8ac2dfe4afb4865"} Jul 24 15:05:00.702808 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:05:00.702502 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:05:00.719594 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:05:00.719546 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" podStartSLOduration=1.818932164 podStartE2EDuration="8.719531467s" totalImagesPullingTime="6.900599303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:05:00.718059278 +0000 UTC m=+519.665552897" watchObservedRunningTime="2026-07-24 15:05:00.719531467 +0000 UTC m=+519.667025052" Jul 24 15:05:06.735835 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:05:06.735807 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kpqp4" Jul 24 15:06:14.447103 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.447071 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c"] Jul 24 15:06:14.450323 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.450304 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.453467 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.453445 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 15:06:14.454535 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.454517 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 15:06:14.454636 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.454532 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mx8dd\"" Jul 24 15:06:14.454636 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.454535 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 15:06:14.462115 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.462092 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c"] Jul 24 15:06:14.499517 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.499487 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.499682 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.499534 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb96x\" (UniqueName: \"kubernetes.io/projected/a359727e-e692-46b0-8beb-ac46a99c4629-kube-api-access-tb96x\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.499682 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.499558 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.499682 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.499635 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.499682 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.499677 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a359727e-e692-46b0-8beb-ac46a99c4629-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.499888 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.499791 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.600498 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.600453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a359727e-e692-46b0-8beb-ac46a99c4629-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.600677 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.600515 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.600677 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.600550 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.600677 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.600592 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb96x\" (UniqueName: \"kubernetes.io/projected/a359727e-e692-46b0-8beb-ac46a99c4629-kube-api-access-tb96x\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.600797 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.600726 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.600797 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.600784 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.600947 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.600929 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.601074 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.601048 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.601227 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.601096 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.603245 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.603221 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a359727e-e692-46b0-8beb-ac46a99c4629-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.603343 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.603272 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a359727e-e692-46b0-8beb-ac46a99c4629-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.609010 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.608991 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb96x\" (UniqueName: \"kubernetes.io/projected/a359727e-e692-46b0-8beb-ac46a99c4629-kube-api-access-tb96x\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fgk6c\" (UID: \"a359727e-e692-46b0-8beb-ac46a99c4629\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.760578 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.760482 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:14.913294 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:14.913262 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c"] Jul 24 15:06:14.916489 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:06:14.916462 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda359727e_e692_46b0_8beb_ac46a99c4629.slice/crio-5ee27b45e806e83c8fff75b255f4ad22e2abd71ee1295eda63ccb0d86ea17ce5 WatchSource:0}: Error finding container 5ee27b45e806e83c8fff75b255f4ad22e2abd71ee1295eda63ccb0d86ea17ce5: Status 404 returned error can't find the container with id 5ee27b45e806e83c8fff75b255f4ad22e2abd71ee1295eda63ccb0d86ea17ce5 Jul 24 15:06:15.006998 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:15.006963 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" event={"ID":"a359727e-e692-46b0-8beb-ac46a99c4629","Type":"ContainerStarted","Data":"5ee27b45e806e83c8fff75b255f4ad22e2abd71ee1295eda63ccb0d86ea17ce5"} Jul 24 15:06:16.014259 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:16.014223 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" event={"ID":"a359727e-e692-46b0-8beb-ac46a99c4629","Type":"ContainerStarted","Data":"c5b61197f77c5b0aba7419e37c5ac9a5a9d600c13bdc9f03d5d6e5e853692861"} Jul 24 15:06:16.038129 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:16.038073 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" podStartSLOduration=1.031335677 podStartE2EDuration="2.038053187s" totalImagesPullingTime="1.00671751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:06:16.037735218 +0000 UTC m=+594.985228814" watchObservedRunningTime="2026-07-24 15:06:16.038053187 +0000 UTC m=+594.985546776" Jul 24 15:06:24.760924 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:24.760877 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:24.763116 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:24.760947 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:24.773510 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:24.773486 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:06:25.061277 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:06:25.061172 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fgk6c" Jul 24 15:09:47.483882 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.483801 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d8bd4bd76-gbz8v"] Jul 24 15:09:47.486869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.486854 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:09:47.489703 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.489682 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 15:09:47.490794 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.490776 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 15:09:47.490860 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.490838 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v7hkw\"" Jul 24 15:09:47.497672 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.497651 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bd4bd76-gbz8v"] Jul 24 15:09:47.614312 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.614282 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bafad5-c645-4048-b34d-7c05ba8163f8-webhook-cert\") pod \"maas-controller-7d8bd4bd76-gbz8v\" (UID: \"a5bafad5-c645-4048-b34d-7c05ba8163f8\") " pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:09:47.614486 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.614336 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5gnh\" (UniqueName: \"kubernetes.io/projected/a5bafad5-c645-4048-b34d-7c05ba8163f8-kube-api-access-s5gnh\") pod \"maas-controller-7d8bd4bd76-gbz8v\" (UID: \"a5bafad5-c645-4048-b34d-7c05ba8163f8\") " pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:09:47.715602 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.715567 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5gnh\" (UniqueName: \"kubernetes.io/projected/a5bafad5-c645-4048-b34d-7c05ba8163f8-kube-api-access-s5gnh\") pod \"maas-controller-7d8bd4bd76-gbz8v\" (UID: \"a5bafad5-c645-4048-b34d-7c05ba8163f8\") " pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:09:47.715743 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.715650 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bafad5-c645-4048-b34d-7c05ba8163f8-webhook-cert\") pod \"maas-controller-7d8bd4bd76-gbz8v\" (UID: \"a5bafad5-c645-4048-b34d-7c05ba8163f8\") " pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:09:47.718257 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.718220 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bafad5-c645-4048-b34d-7c05ba8163f8-webhook-cert\") pod \"maas-controller-7d8bd4bd76-gbz8v\" (UID: \"a5bafad5-c645-4048-b34d-7c05ba8163f8\") " pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:09:47.726288 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.726261 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5gnh\" (UniqueName: \"kubernetes.io/projected/a5bafad5-c645-4048-b34d-7c05ba8163f8-kube-api-access-s5gnh\") pod \"maas-controller-7d8bd4bd76-gbz8v\" (UID: \"a5bafad5-c645-4048-b34d-7c05ba8163f8\") " pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:09:47.797392 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.797303 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:09:47.931881 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.931853 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bd4bd76-gbz8v"] Jul 24 15:09:47.934245 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:09:47.934221 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bafad5_c645_4048_b34d_7c05ba8163f8.slice/crio-7306aab538909c7c4ff823cfbca3160d1c4a1ad4cae71ead924b81729fc47204 WatchSource:0}: Error finding container 7306aab538909c7c4ff823cfbca3160d1c4a1ad4cae71ead924b81729fc47204: Status 404 returned error can't find the container with id 7306aab538909c7c4ff823cfbca3160d1c4a1ad4cae71ead924b81729fc47204 Jul 24 15:09:47.935432 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:47.935414 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:09:48.909444 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:48.909369 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" event={"ID":"a5bafad5-c645-4048-b34d-7c05ba8163f8","Type":"ContainerStarted","Data":"7306aab538909c7c4ff823cfbca3160d1c4a1ad4cae71ead924b81729fc47204"} Jul 24 15:09:50.919260 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:50.919220 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" event={"ID":"a5bafad5-c645-4048-b34d-7c05ba8163f8","Type":"ContainerStarted","Data":"7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0"} Jul 24 15:09:50.919632 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:50.919382 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:09:50.940340 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:09:50.940297 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" podStartSLOduration=1.6499969239999999 podStartE2EDuration="3.940281112s" totalImagesPullingTime="2.290284188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:50.938752759 +0000 UTC m=+809.886246344" watchObservedRunningTime="2026-07-24 15:09:50.940281112 +0000 UTC m=+809.887774697" Jul 24 15:10:01.926022 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:01.925992 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:10:06.662366 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.662325 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8bd4bd76-gbz8v"] Jul 24 15:10:06.662847 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.662581 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" podUID="a5bafad5-c645-4048-b34d-7c05ba8163f8" containerName="manager" containerID="cri-o://7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0" gracePeriod=30 Jul 24 15:10:06.912090 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.912065 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:10:06.986300 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.986219 2557 generic.go:356] "Generic (PLEG): container finished" podID="a5bafad5-c645-4048-b34d-7c05ba8163f8" containerID="7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0" exitCode=0 Jul 24 15:10:06.986300 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.986267 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" Jul 24 15:10:06.986471 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.986302 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" event={"ID":"a5bafad5-c645-4048-b34d-7c05ba8163f8","Type":"ContainerDied","Data":"7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0"} Jul 24 15:10:06.986471 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.986361 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bd4bd76-gbz8v" event={"ID":"a5bafad5-c645-4048-b34d-7c05ba8163f8","Type":"ContainerDied","Data":"7306aab538909c7c4ff823cfbca3160d1c4a1ad4cae71ead924b81729fc47204"} Jul 24 15:10:06.986471 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.986383 2557 scope.go:117] "RemoveContainer" containerID="7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0" Jul 24 15:10:06.994254 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.994236 2557 scope.go:117] "RemoveContainer" containerID="7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0" Jul 24 15:10:06.994483 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:10:06.994463 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0\": container with ID starting with 7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0 not found: ID does not exist" containerID="7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0" Jul 24 15:10:06.994547 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:06.994490 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0"} err="failed to get container status \"7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0\": rpc error: code = NotFound desc = could not find container \"7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0\": container with ID starting with 7b07e4831eabea2f504004167da8c756b5797f627fbe4763dc630dc43d0959f0 not found: ID does not exist" Jul 24 15:10:07.087530 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:07.087497 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bafad5-c645-4048-b34d-7c05ba8163f8-webhook-cert\") pod \"a5bafad5-c645-4048-b34d-7c05ba8163f8\" (UID: \"a5bafad5-c645-4048-b34d-7c05ba8163f8\") " Jul 24 15:10:07.087704 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:07.087603 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5gnh\" (UniqueName: \"kubernetes.io/projected/a5bafad5-c645-4048-b34d-7c05ba8163f8-kube-api-access-s5gnh\") pod \"a5bafad5-c645-4048-b34d-7c05ba8163f8\" (UID: \"a5bafad5-c645-4048-b34d-7c05ba8163f8\") " Jul 24 15:10:07.089887 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:07.089859 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bafad5-c645-4048-b34d-7c05ba8163f8-kube-api-access-s5gnh" (OuterVolumeSpecName: "kube-api-access-s5gnh") pod "a5bafad5-c645-4048-b34d-7c05ba8163f8" (UID: "a5bafad5-c645-4048-b34d-7c05ba8163f8"). InnerVolumeSpecName "kube-api-access-s5gnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:10:07.090007 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:07.089881 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bafad5-c645-4048-b34d-7c05ba8163f8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a5bafad5-c645-4048-b34d-7c05ba8163f8" (UID: "a5bafad5-c645-4048-b34d-7c05ba8163f8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:10:07.188981 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:07.188933 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5gnh\" (UniqueName: \"kubernetes.io/projected/a5bafad5-c645-4048-b34d-7c05ba8163f8-kube-api-access-s5gnh\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:10:07.188981 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:07.188976 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bafad5-c645-4048-b34d-7c05ba8163f8-webhook-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:10:07.313391 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:07.313359 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8bd4bd76-gbz8v"] Jul 24 15:10:07.318423 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:07.318398 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d8bd4bd76-gbz8v"] Jul 24 15:10:07.587346 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:07.587267 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5bafad5-c645-4048-b34d-7c05ba8163f8" path="/var/lib/kubelet/pods/a5bafad5-c645-4048-b34d-7c05ba8163f8/volumes" Jul 24 15:10:08.228322 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.228287 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d8bd4bd76-z5rrt"] Jul 24 15:10:08.228727 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.228594 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5bafad5-c645-4048-b34d-7c05ba8163f8" containerName="manager" Jul 24 15:10:08.228727 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.228605 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5bafad5-c645-4048-b34d-7c05ba8163f8" containerName="manager" Jul 24 15:10:08.228727 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.228656 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5bafad5-c645-4048-b34d-7c05ba8163f8" containerName="manager" Jul 24 15:10:08.232941 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.232924 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:08.236708 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.236661 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 15:10:08.236708 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.236683 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 15:10:08.236914 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.236685 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v7hkw\"" Jul 24 15:10:08.248760 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.248738 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bd4bd76-z5rrt"] Jul 24 15:10:08.296567 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.296532 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcwpk\" (UniqueName: \"kubernetes.io/projected/c229da9c-96e7-4655-85cc-2ea51565f23d-kube-api-access-dcwpk\") pod \"maas-controller-7d8bd4bd76-z5rrt\" (UID: \"c229da9c-96e7-4655-85cc-2ea51565f23d\") " pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:08.296734 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.296575 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c229da9c-96e7-4655-85cc-2ea51565f23d-webhook-cert\") pod \"maas-controller-7d8bd4bd76-z5rrt\" (UID: \"c229da9c-96e7-4655-85cc-2ea51565f23d\") " pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:08.397385 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.397352 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c229da9c-96e7-4655-85cc-2ea51565f23d-webhook-cert\") pod \"maas-controller-7d8bd4bd76-z5rrt\" (UID: \"c229da9c-96e7-4655-85cc-2ea51565f23d\") " pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:08.397497 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.397440 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcwpk\" (UniqueName: \"kubernetes.io/projected/c229da9c-96e7-4655-85cc-2ea51565f23d-kube-api-access-dcwpk\") pod \"maas-controller-7d8bd4bd76-z5rrt\" (UID: \"c229da9c-96e7-4655-85cc-2ea51565f23d\") " pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:08.399911 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.399894 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c229da9c-96e7-4655-85cc-2ea51565f23d-webhook-cert\") pod \"maas-controller-7d8bd4bd76-z5rrt\" (UID: \"c229da9c-96e7-4655-85cc-2ea51565f23d\") " pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:08.407644 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.407611 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcwpk\" (UniqueName: \"kubernetes.io/projected/c229da9c-96e7-4655-85cc-2ea51565f23d-kube-api-access-dcwpk\") pod \"maas-controller-7d8bd4bd76-z5rrt\" (UID: \"c229da9c-96e7-4655-85cc-2ea51565f23d\") " pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:08.542495 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.542405 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:08.685795 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.685762 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bd4bd76-z5rrt"] Jul 24 15:10:08.689047 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:10:08.689019 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc229da9c_96e7_4655_85cc_2ea51565f23d.slice/crio-b52cc25ce7dcc5ff9788e3579947e54de1f29d27fa7e76d43dbf288d158bf952 WatchSource:0}: Error finding container b52cc25ce7dcc5ff9788e3579947e54de1f29d27fa7e76d43dbf288d158bf952: Status 404 returned error can't find the container with id b52cc25ce7dcc5ff9788e3579947e54de1f29d27fa7e76d43dbf288d158bf952 Jul 24 15:10:08.997269 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:08.997236 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" event={"ID":"c229da9c-96e7-4655-85cc-2ea51565f23d","Type":"ContainerStarted","Data":"b52cc25ce7dcc5ff9788e3579947e54de1f29d27fa7e76d43dbf288d158bf952"} Jul 24 15:10:10.002656 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:10.002614 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" event={"ID":"c229da9c-96e7-4655-85cc-2ea51565f23d","Type":"ContainerStarted","Data":"693d9af27f72911df60b1ced039195f0f64124dc85cf3af0abb1c2430ffb48a0"} Jul 24 15:10:10.003034 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:10.002706 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:21.009147 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:21.009116 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" Jul 24 15:10:21.030622 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:10:21.030562 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d8bd4bd76-z5rrt" podStartSLOduration=12.367360816 podStartE2EDuration="13.030543941s" totalImagesPullingTime="663.183125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:10.034387258 +0000 UTC m=+828.981880865" watchObservedRunningTime="2026-07-24 15:10:21.030543941 +0000 UTC m=+839.978037528" Jul 24 15:15:00.159644 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:00.159602 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq"] Jul 24 15:15:00.162923 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:00.162905 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq" Jul 24 15:15:00.166849 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:00.166820 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:15:00.172457 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:00.172436 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq" Jul 24 15:15:00.190466 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:00.190431 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq"] Jul 24 15:15:00.304923 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:00.304901 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq"] Jul 24 15:15:00.307265 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:15:00.307230 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608a814d_d262_475e_9fed_9ebdb29dccee.slice/crio-8651f0d5b80965751ae975983acf6745650376f2338d4ea20ea37a6f35e13336 WatchSource:0}: Error finding container 8651f0d5b80965751ae975983acf6745650376f2338d4ea20ea37a6f35e13336: Status 404 returned error can't find the container with id 8651f0d5b80965751ae975983acf6745650376f2338d4ea20ea37a6f35e13336 Jul 24 15:15:00.309254 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:00.309234 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:15:01.221547 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:01.221505 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq" event={"ID":"608a814d-d262-475e-9fed-9ebdb29dccee","Type":"ContainerStarted","Data":"8651f0d5b80965751ae975983acf6745650376f2338d4ea20ea37a6f35e13336"} Jul 24 15:15:03.232283 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:03.232251 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq" event={"ID":"608a814d-d262-475e-9fed-9ebdb29dccee","Type":"ContainerStarted","Data":"585888cb4f42dc95a78152a86703ee6d7ef073f2e61196140f0b5f9f872be4e9"} Jul 24 15:15:03.249552 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:03.249503 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq" podStartSLOduration=1.073728679 podStartE2EDuration="3.249488717s" totalImagesPullingTime="2.175760038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:15:03.24759051 +0000 UTC m=+1122.195084096" watchObservedRunningTime="2026-07-24 15:15:03.249488717 +0000 UTC m=+1122.196982344" Jul 24 15:15:04.238339 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:04.238301 2557 generic.go:356] "Generic (PLEG): container finished" podID="608a814d-d262-475e-9fed-9ebdb29dccee" containerID="585888cb4f42dc95a78152a86703ee6d7ef073f2e61196140f0b5f9f872be4e9" exitCode=0 Jul 24 15:15:04.238704 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:04.238370 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq" event={"ID":"608a814d-d262-475e-9fed-9ebdb29dccee","Type":"ContainerDied","Data":"585888cb4f42dc95a78152a86703ee6d7ef073f2e61196140f0b5f9f872be4e9"} Jul 24 15:15:05.376199 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:05.376165 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq" Jul 24 15:15:06.249322 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:06.249290 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq" Jul 24 15:15:06.249497 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:06.249325 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748435-h6rxq" event={"ID":"608a814d-d262-475e-9fed-9ebdb29dccee","Type":"ContainerDied","Data":"8651f0d5b80965751ae975983acf6745650376f2338d4ea20ea37a6f35e13336"} Jul 24 15:15:06.249497 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:15:06.249377 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8651f0d5b80965751ae975983acf6745650376f2338d4ea20ea37a6f35e13336" Jul 24 15:23:12.466362 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.466274 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc"] Jul 24 15:23:12.466821 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.466662 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="608a814d-d262-475e-9fed-9ebdb29dccee" containerName="cleanup" Jul 24 15:23:12.466821 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.466675 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="608a814d-d262-475e-9fed-9ebdb29dccee" containerName="cleanup" Jul 24 15:23:12.466821 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.466745 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="608a814d-d262-475e-9fed-9ebdb29dccee" containerName="cleanup" Jul 24 15:23:12.469663 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.469640 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.472433 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.472410 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e35d6958-openshift-default-dockercfg-7gd8t\"" Jul 24 15:23:12.496824 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.496794 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc"] Jul 24 15:23:12.571163 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.571132 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e991568d-a3f5-490c-bac0-ef60113700ec-istiod-ca-cert\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.571384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.571173 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-credential-socket\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.571384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.571209 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-data\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.571384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.571319 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-socket\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.571384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.571346 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e991568d-a3f5-490c-bac0-ef60113700ec-istio-podinfo\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.571562 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.571384 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-envoy\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.571562 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.571438 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-istio-token\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.571562 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.571470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lsc4\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-kube-api-access-9lsc4\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.571716 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.571571 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-certs\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672046 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672005 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-socket\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672046 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672044 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e991568d-a3f5-490c-bac0-ef60113700ec-istio-podinfo\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672060 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-envoy\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672083 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-istio-token\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672101 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lsc4\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-kube-api-access-9lsc4\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672133 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-certs\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672214 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e991568d-a3f5-490c-bac0-ef60113700ec-istiod-ca-cert\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672241 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-credential-socket\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672262 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-data\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672676 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672505 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-socket\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672829 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672804 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-credential-socket\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672874 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672827 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-certs\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.672971 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.672955 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e991568d-a3f5-490c-bac0-ef60113700ec-istiod-ca-cert\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.673064 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.673045 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-data\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.674640 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.674618 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-envoy\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.674853 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.674832 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e991568d-a3f5-490c-bac0-ef60113700ec-istio-podinfo\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.681658 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.681634 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-istio-token\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.681789 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.681673 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lsc4\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-kube-api-access-9lsc4\") pod \"e2e-ait-e35d6958-openshift-default-587f8786b-cndtc\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.779670 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.779559 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:12.919823 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.919797 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc"] Jul 24 15:23:12.921975 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:23:12.921938 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode991568d_a3f5_490c_bac0_ef60113700ec.slice/crio-6b7ebbd41a9b2a303fdc8cb696c4a02ebc3c84c955ab06d5c774b746e790df0e WatchSource:0}: Error finding container 6b7ebbd41a9b2a303fdc8cb696c4a02ebc3c84c955ab06d5c774b746e790df0e: Status 404 returned error can't find the container with id 6b7ebbd41a9b2a303fdc8cb696c4a02ebc3c84c955ab06d5c774b746e790df0e Jul 24 15:23:12.924270 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.924238 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:23:12.924358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.924304 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:23:12.924358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:12.924333 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:23:13.307798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:13.307758 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" event={"ID":"e991568d-a3f5-490c-bac0-ef60113700ec","Type":"ContainerStarted","Data":"a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2"} Jul 24 15:23:13.307977 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:13.307827 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" event={"ID":"e991568d-a3f5-490c-bac0-ef60113700ec","Type":"ContainerStarted","Data":"6b7ebbd41a9b2a303fdc8cb696c4a02ebc3c84c955ab06d5c774b746e790df0e"} Jul 24 15:23:13.329596 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:13.329540 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" podStartSLOduration=1.3295201730000001 podStartE2EDuration="1.329520173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:23:13.327907491 +0000 UTC m=+1612.275401091" watchObservedRunningTime="2026-07-24 15:23:13.329520173 +0000 UTC m=+1612.277013758" Jul 24 15:23:13.780010 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:13.779969 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:13.781441 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:13.781410 2557 patch_prober.go:28] interesting pod/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 24 15:23:13.781573 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:13.781470 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 24 15:23:14.785156 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.785111 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:14.800923 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.800891 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp"] Jul 24 15:23:14.804236 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.804207 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:14.807110 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.807078 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n8r5x\"" Jul 24 15:23:14.807243 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.807203 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e35d6958-serving-cert\"" Jul 24 15:23:14.819684 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.819649 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp"] Jul 24 15:23:14.893377 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.893344 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4530fd2-808a-4fad-8ac8-94d5d9106f71-maas-api-tls\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:14.893377 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.893380 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4530fd2-808a-4fad-8ac8-94d5d9106f71-openshift-service-ca\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:14.893608 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.893415 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffnx\" (UniqueName: \"kubernetes.io/projected/e4530fd2-808a-4fad-8ac8-94d5d9106f71-kube-api-access-cffnx\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:14.994536 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.994489 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4530fd2-808a-4fad-8ac8-94d5d9106f71-maas-api-tls\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:14.994536 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.994536 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4530fd2-808a-4fad-8ac8-94d5d9106f71-openshift-service-ca\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:14.994805 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:23:14.994638 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-serving-cert: secret "maas-api-e2e-ait-e35d6958-serving-cert" not found Jul 24 15:23:14.994805 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.994668 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffnx\" (UniqueName: \"kubernetes.io/projected/e4530fd2-808a-4fad-8ac8-94d5d9106f71-kube-api-access-cffnx\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:14.994805 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:23:14.994705 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4530fd2-808a-4fad-8ac8-94d5d9106f71-maas-api-tls podName:e4530fd2-808a-4fad-8ac8-94d5d9106f71 nodeName:}" failed. No retries permitted until 2026-07-24 15:23:15.494687986 +0000 UTC m=+1614.442181550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4530fd2-808a-4fad-8ac8-94d5d9106f71-maas-api-tls") pod "maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" (UID: "e4530fd2-808a-4fad-8ac8-94d5d9106f71") : secret "maas-api-e2e-ait-e35d6958-serving-cert" not found Jul 24 15:23:14.995154 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:14.995138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4530fd2-808a-4fad-8ac8-94d5d9106f71-openshift-service-ca\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:15.004755 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:15.004731 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffnx\" (UniqueName: \"kubernetes.io/projected/e4530fd2-808a-4fad-8ac8-94d5d9106f71-kube-api-access-cffnx\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:15.316624 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:15.316592 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:15.317533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:15.317509 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:15.498949 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:15.498899 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4530fd2-808a-4fad-8ac8-94d5d9106f71-maas-api-tls\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:15.501491 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:15.501470 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4530fd2-808a-4fad-8ac8-94d5d9106f71-maas-api-tls\") pod \"maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:15.715011 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:15.714981 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:15.855675 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:15.855626 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp"] Jul 24 15:23:15.857825 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:23:15.857798 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4530fd2_808a_4fad_8ac8_94d5d9106f71.slice/crio-8bee42a85cabb9ec8b22430e4ae54e371b2c2ac02302caa56405b5e18273f1d8 WatchSource:0}: Error finding container 8bee42a85cabb9ec8b22430e4ae54e371b2c2ac02302caa56405b5e18273f1d8: Status 404 returned error can't find the container with id 8bee42a85cabb9ec8b22430e4ae54e371b2c2ac02302caa56405b5e18273f1d8 Jul 24 15:23:15.859303 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:15.859283 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:23:16.321263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:16.321222 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" event={"ID":"e4530fd2-808a-4fad-8ac8-94d5d9106f71","Type":"ContainerStarted","Data":"8bee42a85cabb9ec8b22430e4ae54e371b2c2ac02302caa56405b5e18273f1d8"} Jul 24 15:23:18.331701 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:18.331657 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" event={"ID":"e4530fd2-808a-4fad-8ac8-94d5d9106f71","Type":"ContainerStarted","Data":"2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05"} Jul 24 15:23:18.332094 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:18.331818 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:18.352200 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:18.352124 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" podStartSLOduration=2.877574412 podStartE2EDuration="4.352107315s" totalImagesPullingTime="1.474532903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:23:18.35076035 +0000 UTC m=+1617.298253964" watchObservedRunningTime="2026-07-24 15:23:18.352107315 +0000 UTC m=+1617.299600932" Jul 24 15:23:24.342014 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:24.341982 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:30.318240 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:30.318208 2557 patch_prober.go:28] interesting pod/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 24 15:23:30.318601 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:30.318260 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 24 15:23:30.385524 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:30.385488 2557 generic.go:356] "Generic (PLEG): container finished" podID="e991568d-a3f5-490c-bac0-ef60113700ec" containerID="a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2" exitCode=0 Jul 24 15:23:30.385686 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:30.385565 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" event={"ID":"e991568d-a3f5-490c-bac0-ef60113700ec","Type":"ContainerDied","Data":"a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2"} Jul 24 15:23:30.385979 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:30.385961 2557 scope.go:117] "RemoveContainer" containerID="a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2" Jul 24 15:23:30.387220 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:30.387166 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:23:30.387302 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:30.387275 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:23:30.387343 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:30.387315 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:23:31.394528 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:31.394479 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" event={"ID":"e991568d-a3f5-490c-bac0-ef60113700ec","Type":"ContainerStarted","Data":"98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092"} Jul 24 15:23:31.780461 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:31.780425 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:31.780754 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:31.780732 2557 patch_prober.go:28] interesting pod/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 24 15:23:31.780809 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:31.780786 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 24 15:23:32.780431 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:32.780391 2557 patch_prober.go:28] interesting pod/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 24 15:23:32.780825 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:32.780447 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 24 15:23:33.780167 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:33.780129 2557 patch_prober.go:28] interesting pod/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 24 15:23:33.780367 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:33.780219 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 24 15:23:34.780719 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:34.780678 2557 patch_prober.go:28] interesting pod/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 24 15:23:34.781100 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:34.780735 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 24 15:23:35.784285 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:35.784248 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:35.784651 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:35.784492 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:35.785048 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:35.785031 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:23:40.374135 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:40.374095 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj"] Jul 24 15:23:40.380263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:40.380244 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" Jul 24 15:23:40.383273 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:40.383233 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:23:40.385441 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:40.385412 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj"] Jul 24 15:23:40.430154 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:40.430116 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c0aaa-a20f-4976-9b4f-34649cfe912e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj\" (UID: \"144c0aaa-a20f-4976-9b4f-34649cfe912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" Jul 24 15:23:40.531036 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:40.530997 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c0aaa-a20f-4976-9b4f-34649cfe912e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj\" (UID: \"144c0aaa-a20f-4976-9b4f-34649cfe912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" Jul 24 15:23:40.531714 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:40.531690 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c0aaa-a20f-4976-9b4f-34649cfe912e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj\" (UID: \"144c0aaa-a20f-4976-9b4f-34649cfe912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" Jul 24 15:23:40.693838 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:40.693801 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" Jul 24 15:23:40.829155 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:40.829131 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj"] Jul 24 15:23:40.831302 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:23:40.831277 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144c0aaa_a20f_4976_9b4f_34649cfe912e.slice/crio-73cb9c527cd2f6031df12b0d2acaf940ae11bd2ab37db37551f9dc629872646d WatchSource:0}: Error finding container 73cb9c527cd2f6031df12b0d2acaf940ae11bd2ab37db37551f9dc629872646d: Status 404 returned error can't find the container with id 73cb9c527cd2f6031df12b0d2acaf940ae11bd2ab37db37551f9dc629872646d Jul 24 15:23:41.444871 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:41.444840 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" event={"ID":"144c0aaa-a20f-4976-9b4f-34649cfe912e","Type":"ContainerStarted","Data":"da798ffb6f42663cdbf7dbaeddcac2b73d2f51be3f82ff1f029d9caef75ccff6"} Jul 24 15:23:41.445258 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:41.444892 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" event={"ID":"144c0aaa-a20f-4976-9b4f-34649cfe912e","Type":"ContainerStarted","Data":"73cb9c527cd2f6031df12b0d2acaf940ae11bd2ab37db37551f9dc629872646d"} Jul 24 15:23:41.463392 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:41.463332 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" podStartSLOduration=1.463314617 podStartE2EDuration="1.463314617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:23:41.461233025 +0000 UTC m=+1640.408726614" watchObservedRunningTime="2026-07-24 15:23:41.463314617 +0000 UTC m=+1640.410808202" Jul 24 15:23:42.450798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:42.450762 2557 generic.go:356] "Generic (PLEG): container finished" podID="144c0aaa-a20f-4976-9b4f-34649cfe912e" containerID="da798ffb6f42663cdbf7dbaeddcac2b73d2f51be3f82ff1f029d9caef75ccff6" exitCode=0 Jul 24 15:23:42.451222 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:42.450844 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" event={"ID":"144c0aaa-a20f-4976-9b4f-34649cfe912e","Type":"ContainerDied","Data":"da798ffb6f42663cdbf7dbaeddcac2b73d2f51be3f82ff1f029d9caef75ccff6"} Jul 24 15:23:43.586519 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:43.586485 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" Jul 24 15:23:43.655853 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:43.655817 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c0aaa-a20f-4976-9b4f-34649cfe912e-maas-api-service-ca\") pod \"144c0aaa-a20f-4976-9b4f-34649cfe912e\" (UID: \"144c0aaa-a20f-4976-9b4f-34649cfe912e\") " Jul 24 15:23:43.656257 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:43.656236 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144c0aaa-a20f-4976-9b4f-34649cfe912e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "144c0aaa-a20f-4976-9b4f-34649cfe912e" (UID: "144c0aaa-a20f-4976-9b4f-34649cfe912e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:23:43.756638 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:43.756550 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c0aaa-a20f-4976-9b4f-34649cfe912e-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:23:44.460973 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:44.460948 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" Jul 24 15:23:44.460973 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:44.460958 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj" event={"ID":"144c0aaa-a20f-4976-9b4f-34649cfe912e","Type":"ContainerDied","Data":"73cb9c527cd2f6031df12b0d2acaf940ae11bd2ab37db37551f9dc629872646d"} Jul 24 15:23:44.461178 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:44.461007 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73cb9c527cd2f6031df12b0d2acaf940ae11bd2ab37db37551f9dc629872646d" Jul 24 15:23:50.949869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:50.949825 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp"] Jul 24 15:23:50.950357 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:50.950077 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" podUID="e4530fd2-808a-4fad-8ac8-94d5d9106f71" containerName="maas-api" containerID="cri-o://2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05" gracePeriod=30 Jul 24 15:23:51.211917 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.211849 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:51.318547 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.318516 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4530fd2-808a-4fad-8ac8-94d5d9106f71-maas-api-tls\") pod \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " Jul 24 15:23:51.318756 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.318561 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4530fd2-808a-4fad-8ac8-94d5d9106f71-openshift-service-ca\") pod \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " Jul 24 15:23:51.318756 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.318594 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cffnx\" (UniqueName: \"kubernetes.io/projected/e4530fd2-808a-4fad-8ac8-94d5d9106f71-kube-api-access-cffnx\") pod \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\" (UID: \"e4530fd2-808a-4fad-8ac8-94d5d9106f71\") " Jul 24 15:23:51.318932 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.318904 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4530fd2-808a-4fad-8ac8-94d5d9106f71-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4530fd2-808a-4fad-8ac8-94d5d9106f71" (UID: "e4530fd2-808a-4fad-8ac8-94d5d9106f71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:23:51.319062 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.319040 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4530fd2-808a-4fad-8ac8-94d5d9106f71-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:23:51.321171 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.321138 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4530fd2-808a-4fad-8ac8-94d5d9106f71-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4530fd2-808a-4fad-8ac8-94d5d9106f71" (UID: "e4530fd2-808a-4fad-8ac8-94d5d9106f71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:23:51.321171 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.321152 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4530fd2-808a-4fad-8ac8-94d5d9106f71-kube-api-access-cffnx" (OuterVolumeSpecName: "kube-api-access-cffnx") pod "e4530fd2-808a-4fad-8ac8-94d5d9106f71" (UID: "e4530fd2-808a-4fad-8ac8-94d5d9106f71"). InnerVolumeSpecName "kube-api-access-cffnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:23:51.420356 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.420317 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4530fd2-808a-4fad-8ac8-94d5d9106f71-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:23:51.420356 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.420351 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffnx\" (UniqueName: \"kubernetes.io/projected/e4530fd2-808a-4fad-8ac8-94d5d9106f71-kube-api-access-cffnx\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:23:51.491986 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.491896 2557 generic.go:356] "Generic (PLEG): container finished" podID="e4530fd2-808a-4fad-8ac8-94d5d9106f71" containerID="2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05" exitCode=0 Jul 24 15:23:51.491986 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.491946 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" Jul 24 15:23:51.492235 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.491980 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" event={"ID":"e4530fd2-808a-4fad-8ac8-94d5d9106f71","Type":"ContainerDied","Data":"2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05"} Jul 24 15:23:51.492235 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.492033 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp" event={"ID":"e4530fd2-808a-4fad-8ac8-94d5d9106f71","Type":"ContainerDied","Data":"8bee42a85cabb9ec8b22430e4ae54e371b2c2ac02302caa56405b5e18273f1d8"} Jul 24 15:23:51.492235 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.492055 2557 scope.go:117] "RemoveContainer" containerID="2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05" Jul 24 15:23:51.503251 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.501824 2557 scope.go:117] "RemoveContainer" containerID="2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05" Jul 24 15:23:51.503251 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:23:51.502604 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05\": container with ID starting with 2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05 not found: ID does not exist" containerID="2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05" Jul 24 15:23:51.503251 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.502641 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05"} err="failed to get container status \"2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05\": rpc error: code = NotFound desc = could not find container \"2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05\": container with ID starting with 2353a3f1935a00a81c43a98d97d463b922f735f3171379fee6b70e4090a12b05 not found: ID does not exist" Jul 24 15:23:51.514585 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.514558 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp"] Jul 24 15:23:51.519795 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.519770 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e35d6958-fbbd75489-qkpwp"] Jul 24 15:23:51.588873 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:51.588838 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4530fd2-808a-4fad-8ac8-94d5d9106f71" path="/var/lib/kubelet/pods/e4530fd2-808a-4fad-8ac8-94d5d9106f71/volumes" Jul 24 15:23:55.463082 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:55.463042 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj"] Jul 24 15:23:55.468482 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:55.468457 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e35d6958-lzrnj"] Jul 24 15:23:55.589561 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:55.589525 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144c0aaa-a20f-4976-9b4f-34649cfe912e" path="/var/lib/kubelet/pods/144c0aaa-a20f-4976-9b4f-34649cfe912e/volumes" Jul 24 15:23:55.610229 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:55.610172 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc"] Jul 24 15:23:55.610458 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:55.610409 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" containerID="cri-o://98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092" gracePeriod=30 Jul 24 15:23:56.678416 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.678381 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv"] Jul 24 15:23:56.678787 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.678705 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4530fd2-808a-4fad-8ac8-94d5d9106f71" containerName="maas-api" Jul 24 15:23:56.678787 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.678716 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4530fd2-808a-4fad-8ac8-94d5d9106f71" containerName="maas-api" Jul 24 15:23:56.678787 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.678739 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144c0aaa-a20f-4976-9b4f-34649cfe912e" containerName="revoke-keys" Jul 24 15:23:56.678787 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.678746 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="144c0aaa-a20f-4976-9b4f-34649cfe912e" containerName="revoke-keys" Jul 24 15:23:56.678910 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.678806 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="144c0aaa-a20f-4976-9b4f-34649cfe912e" containerName="revoke-keys" Jul 24 15:23:56.678910 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.678813 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4530fd2-808a-4fad-8ac8-94d5d9106f71" containerName="maas-api" Jul 24 15:23:56.683023 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.683005 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.686061 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.686040 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a327d057-gw-openshift-default-dockercfg-8p2xw\"" Jul 24 15:23:56.695936 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.695911 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv"] Jul 24 15:23:56.765907 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.765873 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-credential-socket\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.765907 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.765907 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-data\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.765907 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.765928 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-podinfo\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.766253 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.765996 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-certs\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.766253 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.766056 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-envoy\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.766253 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.766077 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx6sb\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-kube-api-access-vx6sb\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.766253 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.766112 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-token\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.766253 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.766150 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istiod-ca-cert\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.766253 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.766211 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-socket\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.867470 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.867428 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-token\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.867649 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.867498 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istiod-ca-cert\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.867649 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.867537 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-socket\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.867649 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.867598 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-credential-socket\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.867649 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.867622 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-data\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.867649 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.867647 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-podinfo\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.867908 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.867684 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-certs\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.867908 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.867739 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-envoy\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.867908 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.867764 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx6sb\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-kube-api-access-vx6sb\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.868082 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.868050 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-socket\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.868155 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.868114 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-data\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.868288 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.868172 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-certs\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.868330 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.868280 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-credential-socket\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.868404 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.868383 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istiod-ca-cert\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.870127 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.870104 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-envoy\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.870357 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.870338 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-podinfo\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.877762 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.877731 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-token\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.877869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.877772 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx6sb\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-kube-api-access-vx6sb\") pod \"e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:56.992802 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:56.992702 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:57.131656 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:57.131622 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv"] Jul 24 15:23:57.133572 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:23:57.133543 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06c2831_7552_4ef2_9da2_4bdb0decaaa7.slice/crio-5ce7828708152419c32cd1845524431860ef50d8503236008a155dfb46ebde52 WatchSource:0}: Error finding container 5ce7828708152419c32cd1845524431860ef50d8503236008a155dfb46ebde52: Status 404 returned error can't find the container with id 5ce7828708152419c32cd1845524431860ef50d8503236008a155dfb46ebde52 Jul 24 15:23:57.136085 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:57.136052 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:23:57.136179 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:57.136121 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:23:57.136179 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:57.136151 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:23:57.522578 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:57.522536 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" event={"ID":"e06c2831-7552-4ef2-9da2-4bdb0decaaa7","Type":"ContainerStarted","Data":"18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff"} Jul 24 15:23:57.522756 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:57.522601 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" event={"ID":"e06c2831-7552-4ef2-9da2-4bdb0decaaa7","Type":"ContainerStarted","Data":"5ce7828708152419c32cd1845524431860ef50d8503236008a155dfb46ebde52"} Jul 24 15:23:57.552307 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:57.552251 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" podStartSLOduration=1.552235033 podStartE2EDuration="1.552235033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:23:57.550573947 +0000 UTC m=+1656.498067533" watchObservedRunningTime="2026-07-24 15:23:57.552235033 +0000 UTC m=+1656.499728619" Jul 24 15:23:57.993636 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:57.993601 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:58.998115 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:58.998085 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:59.146884 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.146849 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8"] Jul 24 15:23:59.150940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.150916 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.153760 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.153738 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a327d057-serving-cert\"" Jul 24 15:23:59.153901 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.153824 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n8r5x\"" Jul 24 15:23:59.160606 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.160584 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8"] Jul 24 15:23:59.189040 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.189005 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ab460-4452-4e8c-b34b-5a2d48196dca-openshift-service-ca\") pod \"maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.189229 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.189046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ab460-4452-4e8c-b34b-5a2d48196dca-maas-api-tls\") pod \"maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.189229 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.189171 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbj7\" (UniqueName: \"kubernetes.io/projected/a19ab460-4452-4e8c-b34b-5a2d48196dca-kube-api-access-kpbj7\") pod \"maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.290767 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.290684 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbj7\" (UniqueName: \"kubernetes.io/projected/a19ab460-4452-4e8c-b34b-5a2d48196dca-kube-api-access-kpbj7\") pod \"maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.290767 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.290726 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ab460-4452-4e8c-b34b-5a2d48196dca-openshift-service-ca\") pod \"maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.290767 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.290749 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ab460-4452-4e8c-b34b-5a2d48196dca-maas-api-tls\") pod \"maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.291447 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.291428 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ab460-4452-4e8c-b34b-5a2d48196dca-openshift-service-ca\") pod \"maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.293362 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.293340 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ab460-4452-4e8c-b34b-5a2d48196dca-maas-api-tls\") pod \"maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.301247 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.301225 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbj7\" (UniqueName: \"kubernetes.io/projected/a19ab460-4452-4e8c-b34b-5a2d48196dca-kube-api-access-kpbj7\") pod \"maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.462924 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.462888 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:23:59.535330 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.535302 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:59.536204 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.536164 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:23:59.622686 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:23:59.622662 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8"] Jul 24 15:23:59.625206 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:23:59.625156 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19ab460_4452_4e8c_b34b_5a2d48196dca.slice/crio-a4e23fe87aceb1800f3c59bc9be1bf260a5dcd493512e18eebea9cb8300aed6b WatchSource:0}: Error finding container a4e23fe87aceb1800f3c59bc9be1bf260a5dcd493512e18eebea9cb8300aed6b: Status 404 returned error can't find the container with id a4e23fe87aceb1800f3c59bc9be1bf260a5dcd493512e18eebea9cb8300aed6b Jul 24 15:24:00.541222 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.541163 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" event={"ID":"a19ab460-4452-4e8c-b34b-5a2d48196dca","Type":"ContainerStarted","Data":"fada6f7a846cff123a5a45bdcc0d10a65dd54fb9e3d50bc2af06e38a45f7bec7"} Jul 24 15:24:00.541634 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.541259 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:24:00.541634 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.541316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" event={"ID":"a19ab460-4452-4e8c-b34b-5a2d48196dca","Type":"ContainerStarted","Data":"a4e23fe87aceb1800f3c59bc9be1bf260a5dcd493512e18eebea9cb8300aed6b"} Jul 24 15:24:00.562530 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.562470 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" podStartSLOduration=1.20733831 podStartE2EDuration="1.562453234s" totalImagesPullingTime="355.114924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:24:00.561635295 +0000 UTC m=+1659.509128883" watchObservedRunningTime="2026-07-24 15:24:00.562453234 +0000 UTC m=+1659.509946820" Jul 24 15:24:00.860052 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.860028 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:24:00.905103 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905068 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-certs\") pod \"e991568d-a3f5-490c-bac0-ef60113700ec\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " Jul 24 15:24:00.905290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905119 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-credential-socket\") pod \"e991568d-a3f5-490c-bac0-ef60113700ec\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " Jul 24 15:24:00.905290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905137 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-data\") pod \"e991568d-a3f5-490c-bac0-ef60113700ec\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " Jul 24 15:24:00.905290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905169 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-envoy\") pod \"e991568d-a3f5-490c-bac0-ef60113700ec\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " Jul 24 15:24:00.905290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905213 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-istio-token\") pod \"e991568d-a3f5-490c-bac0-ef60113700ec\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " Jul 24 15:24:00.905290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905234 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lsc4\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-kube-api-access-9lsc4\") pod \"e991568d-a3f5-490c-bac0-ef60113700ec\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " Jul 24 15:24:00.905290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905273 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-socket\") pod \"e991568d-a3f5-490c-bac0-ef60113700ec\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " Jul 24 15:24:00.906077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905344 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e991568d-a3f5-490c-bac0-ef60113700ec-istiod-ca-cert\") pod \"e991568d-a3f5-490c-bac0-ef60113700ec\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " Jul 24 15:24:00.906077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905384 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e991568d-a3f5-490c-bac0-ef60113700ec-istio-podinfo\") pod \"e991568d-a3f5-490c-bac0-ef60113700ec\" (UID: \"e991568d-a3f5-490c-bac0-ef60113700ec\") " Jul 24 15:24:00.906077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905414 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e991568d-a3f5-490c-bac0-ef60113700ec" (UID: "e991568d-a3f5-490c-bac0-ef60113700ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:00.906077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905442 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e991568d-a3f5-490c-bac0-ef60113700ec" (UID: "e991568d-a3f5-490c-bac0-ef60113700ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:00.906077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905539 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e991568d-a3f5-490c-bac0-ef60113700ec" (UID: "e991568d-a3f5-490c-bac0-ef60113700ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:00.906077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905687 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:00.906077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905706 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:00.906077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905721 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:00.906077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.905729 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991568d-a3f5-490c-bac0-ef60113700ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e991568d-a3f5-490c-bac0-ef60113700ec" (UID: "e991568d-a3f5-490c-bac0-ef60113700ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:24:00.906460 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.906118 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "e991568d-a3f5-490c-bac0-ef60113700ec" (UID: "e991568d-a3f5-490c-bac0-ef60113700ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:00.908323 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.908299 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e991568d-a3f5-490c-bac0-ef60113700ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e991568d-a3f5-490c-bac0-ef60113700ec" (UID: "e991568d-a3f5-490c-bac0-ef60113700ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:24:00.908323 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.908315 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e991568d-a3f5-490c-bac0-ef60113700ec" (UID: "e991568d-a3f5-490c-bac0-ef60113700ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:00.908611 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.908399 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "e991568d-a3f5-490c-bac0-ef60113700ec" (UID: "e991568d-a3f5-490c-bac0-ef60113700ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:24:00.908611 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:00.908419 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-kube-api-access-9lsc4" (OuterVolumeSpecName: "kube-api-access-9lsc4") pod "e991568d-a3f5-490c-bac0-ef60113700ec" (UID: "e991568d-a3f5-490c-bac0-ef60113700ec"). InnerVolumeSpecName "kube-api-access-9lsc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:24:01.007161 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.007120 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e991568d-a3f5-490c-bac0-ef60113700ec-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:01.007161 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.007155 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e991568d-a3f5-490c-bac0-ef60113700ec-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:01.007161 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.007168 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:01.007499 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.007211 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e991568d-a3f5-490c-bac0-ef60113700ec-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:01.007499 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.007222 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:01.007499 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.007234 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lsc4\" (UniqueName: \"kubernetes.io/projected/e991568d-a3f5-490c-bac0-ef60113700ec-kube-api-access-9lsc4\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:01.547626 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.547593 2557 generic.go:356] "Generic (PLEG): container finished" podID="e991568d-a3f5-490c-bac0-ef60113700ec" containerID="98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092" exitCode=0 Jul 24 15:24:01.548105 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.547662 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" Jul 24 15:24:01.548105 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.547690 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" event={"ID":"e991568d-a3f5-490c-bac0-ef60113700ec","Type":"ContainerDied","Data":"98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092"} Jul 24 15:24:01.548105 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.547780 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc" event={"ID":"e991568d-a3f5-490c-bac0-ef60113700ec","Type":"ContainerDied","Data":"6b7ebbd41a9b2a303fdc8cb696c4a02ebc3c84c955ab06d5c774b746e790df0e"} Jul 24 15:24:01.548105 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.547808 2557 scope.go:117] "RemoveContainer" containerID="98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092" Jul 24 15:24:01.557459 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.557441 2557 scope.go:117] "RemoveContainer" containerID="a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2" Jul 24 15:24:01.565384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.565368 2557 scope.go:117] "RemoveContainer" containerID="98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092" Jul 24 15:24:01.565650 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:24:01.565630 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092\": container with ID starting with 98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092 not found: ID does not exist" containerID="98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092" Jul 24 15:24:01.565706 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.565660 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092"} err="failed to get container status \"98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092\": rpc error: code = NotFound desc = could not find container \"98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092\": container with ID starting with 98ce45fac78a500217b13291ddcef0fa966df787a6aea0fd1b6e6fb3a7cb5092 not found: ID does not exist" Jul 24 15:24:01.565706 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.565678 2557 scope.go:117] "RemoveContainer" containerID="a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2" Jul 24 15:24:01.565933 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:24:01.565913 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2\": container with ID starting with a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2 not found: ID does not exist" containerID="a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2" Jul 24 15:24:01.565985 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.565937 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2"} err="failed to get container status \"a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2\": rpc error: code = NotFound desc = could not find container \"a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2\": container with ID starting with a53ad359fbc5eec3987d5177791594955a8c6b5bca2172be04de8276acd2e6f2 not found: ID does not exist" Jul 24 15:24:01.569900 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.569875 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc"] Jul 24 15:24:01.572690 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.572668 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e35d6958-openshift-default-587f8786b-cndtc"] Jul 24 15:24:01.588028 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:01.588000 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" path="/var/lib/kubelet/pods/e991568d-a3f5-490c-bac0-ef60113700ec/volumes" Jul 24 15:24:06.553805 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:06.553776 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:24:13.606204 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:13.606155 2557 generic.go:356] "Generic (PLEG): container finished" podID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerID="18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff" exitCode=0 Jul 24 15:24:13.606652 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:13.606238 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" event={"ID":"e06c2831-7552-4ef2-9da2-4bdb0decaaa7","Type":"ContainerDied","Data":"18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff"} Jul 24 15:24:13.606700 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:13.606681 2557 scope.go:117] "RemoveContainer" containerID="18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff" Jul 24 15:24:13.607749 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:13.607714 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:24:13.607851 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:13.607785 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:24:13.607851 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:13.607816 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:24:14.536174 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:14.536130 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:24:14.616479 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:14.616441 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" event={"ID":"e06c2831-7552-4ef2-9da2-4bdb0decaaa7","Type":"ContainerStarted","Data":"b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea"} Jul 24 15:24:14.995545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:14.995043 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:24:14.995545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:14.995210 2557 patch_prober.go:28] interesting pod/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 24 15:24:14.995545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:14.995261 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 24 15:24:15.993872 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:15.993832 2557 patch_prober.go:28] interesting pod/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 24 15:24:15.994523 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:15.994481 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 24 15:24:16.994050 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:16.994011 2557 patch_prober.go:28] interesting pod/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 24 15:24:16.995119 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:16.994660 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 24 15:24:17.993378 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:17.993337 2557 patch_prober.go:28] interesting pod/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 24 15:24:17.993676 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:17.993645 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 24 15:24:18.997737 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:18.997705 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:24:18.998172 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:18.997989 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:24:18.998853 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:18.998836 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:24:24.086821 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.086781 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs"] Jul 24 15:24:24.087314 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.087298 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" Jul 24 15:24:24.087386 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.087318 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" Jul 24 15:24:24.087386 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.087331 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" Jul 24 15:24:24.087386 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.087339 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" Jul 24 15:24:24.087530 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.087434 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" Jul 24 15:24:24.091660 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.091634 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" Jul 24 15:24:24.094593 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.094571 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:24:24.102548 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.102515 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs"] Jul 24 15:24:24.224454 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.224416 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0ce47-b7d5-4644-a43f-4aef988ac32c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs\" (UID: \"c7a0ce47-b7d5-4644-a43f-4aef988ac32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" Jul 24 15:24:24.325814 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.325772 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0ce47-b7d5-4644-a43f-4aef988ac32c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs\" (UID: \"c7a0ce47-b7d5-4644-a43f-4aef988ac32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" Jul 24 15:24:24.326516 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.326494 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0ce47-b7d5-4644-a43f-4aef988ac32c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs\" (UID: \"c7a0ce47-b7d5-4644-a43f-4aef988ac32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" Jul 24 15:24:24.403273 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.403241 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" Jul 24 15:24:24.542454 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.542403 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs"] Jul 24 15:24:24.544746 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:24:24.544715 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a0ce47_b7d5_4644_a43f_4aef988ac32c.slice/crio-e4319c0bb9b01d21e1b2fa769f423ce1039b6809a4c7cef3f212bcf1956ffaf3 WatchSource:0}: Error finding container e4319c0bb9b01d21e1b2fa769f423ce1039b6809a4c7cef3f212bcf1956ffaf3: Status 404 returned error can't find the container with id e4319c0bb9b01d21e1b2fa769f423ce1039b6809a4c7cef3f212bcf1956ffaf3 Jul 24 15:24:24.664812 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:24.664709 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" event={"ID":"c7a0ce47-b7d5-4644-a43f-4aef988ac32c","Type":"ContainerStarted","Data":"e4319c0bb9b01d21e1b2fa769f423ce1039b6809a4c7cef3f212bcf1956ffaf3"} Jul 24 15:24:25.671067 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:25.671022 2557 generic.go:356] "Generic (PLEG): container finished" podID="c7a0ce47-b7d5-4644-a43f-4aef988ac32c" containerID="f8db8f0fdd330595b78f55acfed376a7547e52bec22b66493d2bad3ff8d566f6" exitCode=0 Jul 24 15:24:25.671479 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:25.671112 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" event={"ID":"c7a0ce47-b7d5-4644-a43f-4aef988ac32c","Type":"ContainerDied","Data":"f8db8f0fdd330595b78f55acfed376a7547e52bec22b66493d2bad3ff8d566f6"} Jul 24 15:24:26.807891 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:26.807859 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" Jul 24 15:24:26.951326 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:26.951233 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0ce47-b7d5-4644-a43f-4aef988ac32c-maas-api-service-ca\") pod \"c7a0ce47-b7d5-4644-a43f-4aef988ac32c\" (UID: \"c7a0ce47-b7d5-4644-a43f-4aef988ac32c\") " Jul 24 15:24:26.951626 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:26.951600 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a0ce47-b7d5-4644-a43f-4aef988ac32c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7a0ce47-b7d5-4644-a43f-4aef988ac32c" (UID: "c7a0ce47-b7d5-4644-a43f-4aef988ac32c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:24:27.052386 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:27.052351 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0ce47-b7d5-4644-a43f-4aef988ac32c-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:27.681533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:27.681503 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" Jul 24 15:24:27.681709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:27.681502 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs" event={"ID":"c7a0ce47-b7d5-4644-a43f-4aef988ac32c","Type":"ContainerDied","Data":"e4319c0bb9b01d21e1b2fa769f423ce1039b6809a4c7cef3f212bcf1956ffaf3"} Jul 24 15:24:27.681709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:27.681627 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4319c0bb9b01d21e1b2fa769f423ce1039b6809a4c7cef3f212bcf1956ffaf3" Jul 24 15:24:34.556028 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.555996 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8"] Jul 24 15:24:34.556548 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.556235 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" podUID="a19ab460-4452-4e8c-b34b-5a2d48196dca" containerName="maas-api" containerID="cri-o://fada6f7a846cff123a5a45bdcc0d10a65dd54fb9e3d50bc2af06e38a45f7bec7" gracePeriod=30 Jul 24 15:24:34.714748 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.714712 2557 generic.go:356] "Generic (PLEG): container finished" podID="a19ab460-4452-4e8c-b34b-5a2d48196dca" containerID="fada6f7a846cff123a5a45bdcc0d10a65dd54fb9e3d50bc2af06e38a45f7bec7" exitCode=0 Jul 24 15:24:34.714907 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.714786 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" event={"ID":"a19ab460-4452-4e8c-b34b-5a2d48196dca","Type":"ContainerDied","Data":"fada6f7a846cff123a5a45bdcc0d10a65dd54fb9e3d50bc2af06e38a45f7bec7"} Jul 24 15:24:34.811110 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.811020 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:24:34.919435 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.919401 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpbj7\" (UniqueName: \"kubernetes.io/projected/a19ab460-4452-4e8c-b34b-5a2d48196dca-kube-api-access-kpbj7\") pod \"a19ab460-4452-4e8c-b34b-5a2d48196dca\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " Jul 24 15:24:34.919623 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.919467 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ab460-4452-4e8c-b34b-5a2d48196dca-openshift-service-ca\") pod \"a19ab460-4452-4e8c-b34b-5a2d48196dca\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " Jul 24 15:24:34.919623 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.919494 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ab460-4452-4e8c-b34b-5a2d48196dca-maas-api-tls\") pod \"a19ab460-4452-4e8c-b34b-5a2d48196dca\" (UID: \"a19ab460-4452-4e8c-b34b-5a2d48196dca\") " Jul 24 15:24:34.919765 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.919744 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19ab460-4452-4e8c-b34b-5a2d48196dca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a19ab460-4452-4e8c-b34b-5a2d48196dca" (UID: "a19ab460-4452-4e8c-b34b-5a2d48196dca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:24:34.921772 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.921744 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19ab460-4452-4e8c-b34b-5a2d48196dca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a19ab460-4452-4e8c-b34b-5a2d48196dca" (UID: "a19ab460-4452-4e8c-b34b-5a2d48196dca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:24:34.921924 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:34.921912 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19ab460-4452-4e8c-b34b-5a2d48196dca-kube-api-access-kpbj7" (OuterVolumeSpecName: "kube-api-access-kpbj7") pod "a19ab460-4452-4e8c-b34b-5a2d48196dca" (UID: "a19ab460-4452-4e8c-b34b-5a2d48196dca"). InnerVolumeSpecName "kube-api-access-kpbj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:24:35.020491 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:35.020443 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ab460-4452-4e8c-b34b-5a2d48196dca-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:35.020491 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:35.020483 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ab460-4452-4e8c-b34b-5a2d48196dca-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:35.020491 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:35.020496 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpbj7\" (UniqueName: \"kubernetes.io/projected/a19ab460-4452-4e8c-b34b-5a2d48196dca-kube-api-access-kpbj7\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:35.720645 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:35.720609 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" event={"ID":"a19ab460-4452-4e8c-b34b-5a2d48196dca","Type":"ContainerDied","Data":"a4e23fe87aceb1800f3c59bc9be1bf260a5dcd493512e18eebea9cb8300aed6b"} Jul 24 15:24:35.721085 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:35.720675 2557 scope.go:117] "RemoveContainer" containerID="fada6f7a846cff123a5a45bdcc0d10a65dd54fb9e3d50bc2af06e38a45f7bec7" Jul 24 15:24:35.721085 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:35.720619 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8" Jul 24 15:24:35.764537 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:35.764506 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8"] Jul 24 15:24:35.775831 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:35.775795 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a327d057-5bf59cb7b-5jcn8"] Jul 24 15:24:37.590047 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:37.590008 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19ab460-4452-4e8c-b34b-5a2d48196dca" path="/var/lib/kubelet/pods/a19ab460-4452-4e8c-b34b-5a2d48196dca/volumes" Jul 24 15:24:39.175065 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:39.175028 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs"] Jul 24 15:24:39.178487 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:39.178458 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a327d057-v6jqs"] Jul 24 15:24:39.322220 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:39.322164 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv"] Jul 24 15:24:39.322439 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:39.322412 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" containerID="cri-o://b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea" gracePeriod=30 Jul 24 15:24:39.588278 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:39.588165 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a0ce47-b7d5-4644-a43f-4aef988ac32c" path="/var/lib/kubelet/pods/c7a0ce47-b7d5-4644-a43f-4aef988ac32c/volumes" Jul 24 15:24:40.149742 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.149713 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg"] Jul 24 15:24:40.150065 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.150053 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19ab460-4452-4e8c-b34b-5a2d48196dca" containerName="maas-api" Jul 24 15:24:40.150114 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.150069 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19ab460-4452-4e8c-b34b-5a2d48196dca" containerName="maas-api" Jul 24 15:24:40.150114 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.150085 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a0ce47-b7d5-4644-a43f-4aef988ac32c" containerName="revoke-keys" Jul 24 15:24:40.150114 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.150090 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a0ce47-b7d5-4644-a43f-4aef988ac32c" containerName="revoke-keys" Jul 24 15:24:40.150222 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.150149 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a0ce47-b7d5-4644-a43f-4aef988ac32c" containerName="revoke-keys" Jul 24 15:24:40.150222 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.150160 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e991568d-a3f5-490c-bac0-ef60113700ec" containerName="istio-proxy" Jul 24 15:24:40.150222 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.150167 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19ab460-4452-4e8c-b34b-5a2d48196dca" containerName="maas-api" Jul 24 15:24:40.154705 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.154687 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.157755 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.157734 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e2f4813e-openshift-default-dockercfg-qx96v\"" Jul 24 15:24:40.165176 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.165154 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg"] Jul 24 15:24:40.269235 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.269194 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-credential-socket\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.269694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.269262 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-podinfo\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.269694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.269287 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-socket\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.269694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.269308 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istiod-ca-cert\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.269694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.269353 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52kmv\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-kube-api-access-52kmv\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.269694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.269447 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-data\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.269694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.269485 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-token\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.269694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.269540 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-certs\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.269694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.269563 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-envoy\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.370830 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.370788 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-certs\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.370830 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.370830 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-envoy\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371084 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.370854 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-credential-socket\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371084 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.370898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-podinfo\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371084 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.370916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-socket\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371084 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.370937 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istiod-ca-cert\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371084 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.370967 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52kmv\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-kube-api-access-52kmv\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371084 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.371006 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-data\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371084 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.371030 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-token\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371484 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.371316 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-certs\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371484 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.371341 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-credential-socket\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371589 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.371540 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-socket\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371589 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.371551 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-data\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.371731 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.371713 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istiod-ca-cert\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.373794 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.373769 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-envoy\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.373988 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.373971 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-podinfo\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.380092 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.380066 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-token\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.380290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.380267 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52kmv\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-kube-api-access-52kmv\") pod \"e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.465587 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.465552 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:40.632105 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.632077 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg"] Jul 24 15:24:40.634014 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:24:40.633986 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50f44fa_663b_4149_b73a_c9c7dd0664bd.slice/crio-9244e56765193ae5fbc38b45328faa41f2dbfd013939b0c4b0ca67e7f4b753e1 WatchSource:0}: Error finding container 9244e56765193ae5fbc38b45328faa41f2dbfd013939b0c4b0ca67e7f4b753e1: Status 404 returned error can't find the container with id 9244e56765193ae5fbc38b45328faa41f2dbfd013939b0c4b0ca67e7f4b753e1 Jul 24 15:24:40.636335 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.636297 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:24:40.636406 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.636390 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:24:40.636449 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.636430 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:24:40.747533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.747435 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" event={"ID":"e50f44fa-663b-4149-b73a-c9c7dd0664bd","Type":"ContainerStarted","Data":"54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926"} Jul 24 15:24:40.747533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.747497 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" event={"ID":"e50f44fa-663b-4149-b73a-c9c7dd0664bd","Type":"ContainerStarted","Data":"9244e56765193ae5fbc38b45328faa41f2dbfd013939b0c4b0ca67e7f4b753e1"} Jul 24 15:24:40.773391 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:40.773323 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" podStartSLOduration=0.773299732 podStartE2EDuration="773.299732ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:24:40.771955512 +0000 UTC m=+1699.719449099" watchObservedRunningTime="2026-07-24 15:24:40.773299732 +0000 UTC m=+1699.720793318" Jul 24 15:24:41.466400 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:41.466365 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:41.471529 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:41.471502 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:41.751767 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:41.751672 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:41.752692 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:41.752671 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:42.280706 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.280673 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q"] Jul 24 15:24:42.284176 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.284157 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.287007 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.286982 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e2f4813e-serving-cert\"" Jul 24 15:24:42.287123 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.286985 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n8r5x\"" Jul 24 15:24:42.296736 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.296706 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q"] Jul 24 15:24:42.389429 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.389393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53596628-a2ad-4d32-a901-ff1f90cbddc9-maas-api-tls\") pod \"maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.389599 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.389440 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk62m\" (UniqueName: \"kubernetes.io/projected/53596628-a2ad-4d32-a901-ff1f90cbddc9-kube-api-access-tk62m\") pod \"maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.389599 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.389532 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53596628-a2ad-4d32-a901-ff1f90cbddc9-openshift-service-ca\") pod \"maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.490685 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.490577 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53596628-a2ad-4d32-a901-ff1f90cbddc9-openshift-service-ca\") pod \"maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.491062 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.490752 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53596628-a2ad-4d32-a901-ff1f90cbddc9-maas-api-tls\") pod \"maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.491062 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.490787 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk62m\" (UniqueName: \"kubernetes.io/projected/53596628-a2ad-4d32-a901-ff1f90cbddc9-kube-api-access-tk62m\") pod \"maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.491232 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.491175 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53596628-a2ad-4d32-a901-ff1f90cbddc9-openshift-service-ca\") pod \"maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.493268 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.493249 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53596628-a2ad-4d32-a901-ff1f90cbddc9-maas-api-tls\") pod \"maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.499656 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.499631 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk62m\" (UniqueName: \"kubernetes.io/projected/53596628-a2ad-4d32-a901-ff1f90cbddc9-kube-api-access-tk62m\") pod \"maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.595337 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.595239 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:42.735704 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.735666 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q"] Jul 24 15:24:42.737806 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:24:42.737776 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53596628_a2ad_4d32_a901_ff1f90cbddc9.slice/crio-179b3ae4ae0c4e29c3bf619b7204d7f3fb9b26f52c575a7895e0c44c1d8e7ded WatchSource:0}: Error finding container 179b3ae4ae0c4e29c3bf619b7204d7f3fb9b26f52c575a7895e0c44c1d8e7ded: Status 404 returned error can't find the container with id 179b3ae4ae0c4e29c3bf619b7204d7f3fb9b26f52c575a7895e0c44c1d8e7ded Jul 24 15:24:42.757488 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:42.757455 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" event={"ID":"53596628-a2ad-4d32-a901-ff1f90cbddc9","Type":"ContainerStarted","Data":"179b3ae4ae0c4e29c3bf619b7204d7f3fb9b26f52c575a7895e0c44c1d8e7ded"} Jul 24 15:24:43.763589 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:43.763556 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" event={"ID":"53596628-a2ad-4d32-a901-ff1f90cbddc9","Type":"ContainerStarted","Data":"1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a"} Jul 24 15:24:43.764008 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:43.763854 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:43.783324 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:43.783271 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" podStartSLOduration=1.421052903 podStartE2EDuration="1.783255807s" totalImagesPullingTime="362.202904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:24:43.781669803 +0000 UTC m=+1702.729163403" watchObservedRunningTime="2026-07-24 15:24:43.783255807 +0000 UTC m=+1702.730749438" Jul 24 15:24:44.576198 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.576163 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:24:44.610420 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.610348 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-data\") pod \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " Jul 24 15:24:44.610577 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.610432 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-podinfo\") pod \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " Jul 24 15:24:44.610577 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.610457 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-envoy\") pod \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " Jul 24 15:24:44.610577 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.610493 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-token\") pod \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " Jul 24 15:24:44.610577 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.610548 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-credential-socket\") pod \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " Jul 24 15:24:44.611082 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.610619 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-socket\") pod \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " Jul 24 15:24:44.611082 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.610655 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-certs\") pod \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " Jul 24 15:24:44.611082 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.610682 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istiod-ca-cert\") pod \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " Jul 24 15:24:44.611082 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.610727 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx6sb\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-kube-api-access-vx6sb\") pod \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\" (UID: \"e06c2831-7552-4ef2-9da2-4bdb0decaaa7\") " Jul 24 15:24:44.611331 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.611254 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e06c2831-7552-4ef2-9da2-4bdb0decaaa7" (UID: "e06c2831-7552-4ef2-9da2-4bdb0decaaa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:44.611331 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.611268 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-data" (OuterVolumeSpecName: "istio-data") pod "e06c2831-7552-4ef2-9da2-4bdb0decaaa7" (UID: "e06c2831-7552-4ef2-9da2-4bdb0decaaa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:44.611643 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.611557 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e06c2831-7552-4ef2-9da2-4bdb0decaaa7" (UID: "e06c2831-7552-4ef2-9da2-4bdb0decaaa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:24:44.611643 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.611620 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e06c2831-7552-4ef2-9da2-4bdb0decaaa7" (UID: "e06c2831-7552-4ef2-9da2-4bdb0decaaa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:44.611810 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.611719 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e06c2831-7552-4ef2-9da2-4bdb0decaaa7" (UID: "e06c2831-7552-4ef2-9da2-4bdb0decaaa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:44.613123 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.613102 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-kube-api-access-vx6sb" (OuterVolumeSpecName: "kube-api-access-vx6sb") pod "e06c2831-7552-4ef2-9da2-4bdb0decaaa7" (UID: "e06c2831-7552-4ef2-9da2-4bdb0decaaa7"). InnerVolumeSpecName "kube-api-access-vx6sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:24:44.613365 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.613312 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e06c2831-7552-4ef2-9da2-4bdb0decaaa7" (UID: "e06c2831-7552-4ef2-9da2-4bdb0decaaa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:24:44.613487 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.613358 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-token" (OuterVolumeSpecName: "istio-token") pod "e06c2831-7552-4ef2-9da2-4bdb0decaaa7" (UID: "e06c2831-7552-4ef2-9da2-4bdb0decaaa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:24:44.613487 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.613367 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e06c2831-7552-4ef2-9da2-4bdb0decaaa7" (UID: "e06c2831-7552-4ef2-9da2-4bdb0decaaa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:24:44.712333 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.712290 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:44.712333 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.712327 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:44.712333 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.712336 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:44.712333 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.712347 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:44.712665 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.712356 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:44.712665 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.712365 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:44.712665 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.712373 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:44.712665 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.712381 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx6sb\" (UniqueName: \"kubernetes.io/projected/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-kube-api-access-vx6sb\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:44.712665 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.712388 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06c2831-7552-4ef2-9da2-4bdb0decaaa7-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:24:44.770156 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.770117 2557 generic.go:356] "Generic (PLEG): container finished" podID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerID="b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea" exitCode=0 Jul 24 15:24:44.770592 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.770161 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" Jul 24 15:24:44.770592 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.770217 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" event={"ID":"e06c2831-7552-4ef2-9da2-4bdb0decaaa7","Type":"ContainerDied","Data":"b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea"} Jul 24 15:24:44.770592 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.770273 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv" event={"ID":"e06c2831-7552-4ef2-9da2-4bdb0decaaa7","Type":"ContainerDied","Data":"5ce7828708152419c32cd1845524431860ef50d8503236008a155dfb46ebde52"} Jul 24 15:24:44.770592 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.770299 2557 scope.go:117] "RemoveContainer" containerID="b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea" Jul 24 15:24:44.779826 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.779808 2557 scope.go:117] "RemoveContainer" containerID="18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff" Jul 24 15:24:44.787569 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.787552 2557 scope.go:117] "RemoveContainer" containerID="b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea" Jul 24 15:24:44.787834 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:24:44.787817 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea\": container with ID starting with b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea not found: ID does not exist" containerID="b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea" Jul 24 15:24:44.787876 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.787843 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea"} err="failed to get container status \"b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea\": rpc error: code = NotFound desc = could not find container \"b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea\": container with ID starting with b5188680701d83a53f5fe54973b6d86df35da3e2c875a6506b8208f1480515ea not found: ID does not exist" Jul 24 15:24:44.787876 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.787860 2557 scope.go:117] "RemoveContainer" containerID="18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff" Jul 24 15:24:44.788108 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:24:44.788090 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff\": container with ID starting with 18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff not found: ID does not exist" containerID="18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff" Jul 24 15:24:44.788150 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.788114 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff"} err="failed to get container status \"18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff\": rpc error: code = NotFound desc = could not find container \"18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff\": container with ID starting with 18cd4dfbbf949675fb33320425366abf7199e2b40ee670026cd73e07eb48faff not found: ID does not exist" Jul 24 15:24:44.807753 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.807721 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv"] Jul 24 15:24:44.816904 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:44.816878 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a327d057-gw-openshift-default-86b7969c4d-7bfvv"] Jul 24 15:24:45.587349 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:45.587321 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" path="/var/lib/kubelet/pods/e06c2831-7552-4ef2-9da2-4bdb0decaaa7/volumes" Jul 24 15:24:49.776309 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:49.776281 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:24:56.753702 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:56.753620 2557 patch_prober.go:28] interesting pod/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:24:56.753702 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:56.753677 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:24:56.837992 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:56.837902 2557 generic.go:356] "Generic (PLEG): container finished" podID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerID="54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926" exitCode=0 Jul 24 15:24:56.838127 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:56.837979 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" event={"ID":"e50f44fa-663b-4149-b73a-c9c7dd0664bd","Type":"ContainerDied","Data":"54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926"} Jul 24 15:24:56.838423 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:56.838396 2557 scope.go:117] "RemoveContainer" containerID="54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926" Jul 24 15:24:56.839434 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:56.839408 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:24:56.839502 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:56.839487 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:24:56.839544 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:56.839530 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:24:57.844085 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:57.844052 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" event={"ID":"e50f44fa-663b-4149-b73a-c9c7dd0664bd","Type":"ContainerStarted","Data":"b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75"} Jul 24 15:24:58.465812 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:58.465766 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:24:58.466001 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:58.465964 2557 patch_prober.go:28] interesting pod/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:24:58.466048 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:58.466011 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:24:59.466433 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:59.466383 2557 patch_prober.go:28] interesting pod/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:24:59.466852 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:24:59.466449 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:25:00.466529 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:00.466493 2557 patch_prober.go:28] interesting pod/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:25:00.466953 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:00.466548 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:25:01.466147 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:01.466110 2557 patch_prober.go:28] interesting pod/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:25:01.466358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:01.466171 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:25:02.470169 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:02.470133 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:25:02.470545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:02.470409 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:25:02.471172 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:02.471153 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:25:08.016202 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.016144 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc"] Jul 24 15:25:08.016669 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.016646 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" Jul 24 15:25:08.016669 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.016664 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" Jul 24 15:25:08.016780 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.016755 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" Jul 24 15:25:08.016780 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.016772 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" Jul 24 15:25:08.019718 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.019696 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" Jul 24 15:25:08.022765 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.022744 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:25:08.027735 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.027708 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc"] Jul 24 15:25:08.126639 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.126601 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75dce3c9-f188-48dd-80f9-64de937bf33e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc\" (UID: \"75dce3c9-f188-48dd-80f9-64de937bf33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" Jul 24 15:25:08.227676 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.227633 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75dce3c9-f188-48dd-80f9-64de937bf33e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc\" (UID: \"75dce3c9-f188-48dd-80f9-64de937bf33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" Jul 24 15:25:08.228506 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.228480 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75dce3c9-f188-48dd-80f9-64de937bf33e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc\" (UID: \"75dce3c9-f188-48dd-80f9-64de937bf33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" Jul 24 15:25:08.330859 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.330772 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" Jul 24 15:25:08.466038 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.465994 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc"] Jul 24 15:25:08.468384 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:25:08.468357 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75dce3c9_f188_48dd_80f9_64de937bf33e.slice/crio-82af8efc18084c9c51265fdb525e8a0ca64828ec32c201fe1374076b3addfd51 WatchSource:0}: Error finding container 82af8efc18084c9c51265fdb525e8a0ca64828ec32c201fe1374076b3addfd51: Status 404 returned error can't find the container with id 82af8efc18084c9c51265fdb525e8a0ca64828ec32c201fe1374076b3addfd51 Jul 24 15:25:08.900640 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.900538 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" event={"ID":"75dce3c9-f188-48dd-80f9-64de937bf33e","Type":"ContainerStarted","Data":"c437400ce8a15c1dd3a1288c7492e4ae55a26ac8b5109b3cb54d0651b866cccf"} Jul 24 15:25:08.900640 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.900602 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" event={"ID":"75dce3c9-f188-48dd-80f9-64de937bf33e","Type":"ContainerStarted","Data":"82af8efc18084c9c51265fdb525e8a0ca64828ec32c201fe1374076b3addfd51"} Jul 24 15:25:08.918014 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:08.917955 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" podStartSLOduration=1.917938635 podStartE2EDuration="1.917938635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:25:08.91667142 +0000 UTC m=+1727.864165027" watchObservedRunningTime="2026-07-24 15:25:08.917938635 +0000 UTC m=+1727.865432221" Jul 24 15:25:09.906801 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:09.906765 2557 generic.go:356] "Generic (PLEG): container finished" podID="75dce3c9-f188-48dd-80f9-64de937bf33e" containerID="c437400ce8a15c1dd3a1288c7492e4ae55a26ac8b5109b3cb54d0651b866cccf" exitCode=0 Jul 24 15:25:09.907231 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:09.906855 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" event={"ID":"75dce3c9-f188-48dd-80f9-64de937bf33e","Type":"ContainerDied","Data":"c437400ce8a15c1dd3a1288c7492e4ae55a26ac8b5109b3cb54d0651b866cccf"} Jul 24 15:25:11.042078 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:11.042053 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" Jul 24 15:25:11.154580 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:11.154542 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75dce3c9-f188-48dd-80f9-64de937bf33e-maas-api-service-ca\") pod \"75dce3c9-f188-48dd-80f9-64de937bf33e\" (UID: \"75dce3c9-f188-48dd-80f9-64de937bf33e\") " Jul 24 15:25:11.155256 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:11.155220 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75dce3c9-f188-48dd-80f9-64de937bf33e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75dce3c9-f188-48dd-80f9-64de937bf33e" (UID: "75dce3c9-f188-48dd-80f9-64de937bf33e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:25:11.255679 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:11.255592 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75dce3c9-f188-48dd-80f9-64de937bf33e-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:11.916836 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:11.916802 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" event={"ID":"75dce3c9-f188-48dd-80f9-64de937bf33e","Type":"ContainerDied","Data":"82af8efc18084c9c51265fdb525e8a0ca64828ec32c201fe1374076b3addfd51"} Jul 24 15:25:11.916836 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:11.916830 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc" Jul 24 15:25:11.917059 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:11.916854 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82af8efc18084c9c51265fdb525e8a0ca64828ec32c201fe1374076b3addfd51" Jul 24 15:25:18.657337 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.657300 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q"] Jul 24 15:25:18.657800 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.657491 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" podUID="53596628-a2ad-4d32-a901-ff1f90cbddc9" containerName="maas-api" containerID="cri-o://1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a" gracePeriod=30 Jul 24 15:25:18.903780 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.903753 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:25:18.917393 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.917295 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53596628-a2ad-4d32-a901-ff1f90cbddc9-maas-api-tls\") pod \"53596628-a2ad-4d32-a901-ff1f90cbddc9\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " Jul 24 15:25:18.917393 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.917344 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk62m\" (UniqueName: \"kubernetes.io/projected/53596628-a2ad-4d32-a901-ff1f90cbddc9-kube-api-access-tk62m\") pod \"53596628-a2ad-4d32-a901-ff1f90cbddc9\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " Jul 24 15:25:18.917393 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.917385 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53596628-a2ad-4d32-a901-ff1f90cbddc9-openshift-service-ca\") pod \"53596628-a2ad-4d32-a901-ff1f90cbddc9\" (UID: \"53596628-a2ad-4d32-a901-ff1f90cbddc9\") " Jul 24 15:25:18.917972 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.917933 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53596628-a2ad-4d32-a901-ff1f90cbddc9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53596628-a2ad-4d32-a901-ff1f90cbddc9" (UID: "53596628-a2ad-4d32-a901-ff1f90cbddc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:25:18.919638 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.919611 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53596628-a2ad-4d32-a901-ff1f90cbddc9-kube-api-access-tk62m" (OuterVolumeSpecName: "kube-api-access-tk62m") pod "53596628-a2ad-4d32-a901-ff1f90cbddc9" (UID: "53596628-a2ad-4d32-a901-ff1f90cbddc9"). InnerVolumeSpecName "kube-api-access-tk62m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:25:18.919741 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.919675 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53596628-a2ad-4d32-a901-ff1f90cbddc9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53596628-a2ad-4d32-a901-ff1f90cbddc9" (UID: "53596628-a2ad-4d32-a901-ff1f90cbddc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:25:18.949899 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.949869 2557 generic.go:356] "Generic (PLEG): container finished" podID="53596628-a2ad-4d32-a901-ff1f90cbddc9" containerID="1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a" exitCode=0 Jul 24 15:25:18.950095 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.949920 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" event={"ID":"53596628-a2ad-4d32-a901-ff1f90cbddc9","Type":"ContainerDied","Data":"1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a"} Jul 24 15:25:18.950095 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.949924 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" Jul 24 15:25:18.950095 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.949964 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q" event={"ID":"53596628-a2ad-4d32-a901-ff1f90cbddc9","Type":"ContainerDied","Data":"179b3ae4ae0c4e29c3bf619b7204d7f3fb9b26f52c575a7895e0c44c1d8e7ded"} Jul 24 15:25:18.950095 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.950011 2557 scope.go:117] "RemoveContainer" containerID="1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a" Jul 24 15:25:18.959029 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.958994 2557 scope.go:117] "RemoveContainer" containerID="1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a" Jul 24 15:25:18.959306 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:25:18.959285 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a\": container with ID starting with 1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a not found: ID does not exist" containerID="1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a" Jul 24 15:25:18.959377 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.959317 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a"} err="failed to get container status \"1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a\": rpc error: code = NotFound desc = could not find container \"1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a\": container with ID starting with 1ce4ead37a72e1c744cbdf0263097ae7c1c69a5189e571acf05469476cc84c4a not found: ID does not exist" Jul 24 15:25:18.971108 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.971084 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q"] Jul 24 15:25:18.976227 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:18.976201 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2f4813e-6868cd78c8-wd69q"] Jul 24 15:25:19.018165 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:19.018136 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53596628-a2ad-4d32-a901-ff1f90cbddc9-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:19.018165 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:19.018158 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk62m\" (UniqueName: \"kubernetes.io/projected/53596628-a2ad-4d32-a901-ff1f90cbddc9-kube-api-access-tk62m\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:19.018165 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:19.018168 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53596628-a2ad-4d32-a901-ff1f90cbddc9-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:19.587984 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:19.587949 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53596628-a2ad-4d32-a901-ff1f90cbddc9" path="/var/lib/kubelet/pods/53596628-a2ad-4d32-a901-ff1f90cbddc9/volumes" Jul 24 15:25:23.191799 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:23.191764 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc"] Jul 24 15:25:23.198667 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:23.198635 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2f4813e-f26nc"] Jul 24 15:25:23.587763 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:23.587674 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75dce3c9-f188-48dd-80f9-64de937bf33e" path="/var/lib/kubelet/pods/75dce3c9-f188-48dd-80f9-64de937bf33e/volumes" Jul 24 15:25:24.434902 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:24.434870 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg"] Jul 24 15:25:24.435408 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:24.435078 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" containerID="cri-o://b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75" gracePeriod=30 Jul 24 15:25:25.218940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.218912 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv"] Jul 24 15:25:25.219273 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.219258 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75dce3c9-f188-48dd-80f9-64de937bf33e" containerName="revoke-keys" Jul 24 15:25:25.219320 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.219275 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="75dce3c9-f188-48dd-80f9-64de937bf33e" containerName="revoke-keys" Jul 24 15:25:25.219320 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.219288 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" Jul 24 15:25:25.219320 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.219293 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e06c2831-7552-4ef2-9da2-4bdb0decaaa7" containerName="istio-proxy" Jul 24 15:25:25.219320 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.219307 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53596628-a2ad-4d32-a901-ff1f90cbddc9" containerName="maas-api" Jul 24 15:25:25.219320 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.219313 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="53596628-a2ad-4d32-a901-ff1f90cbddc9" containerName="maas-api" Jul 24 15:25:25.219476 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.219382 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="53596628-a2ad-4d32-a901-ff1f90cbddc9" containerName="maas-api" Jul 24 15:25:25.219476 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.219391 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="75dce3c9-f188-48dd-80f9-64de937bf33e" containerName="revoke-keys" Jul 24 15:25:25.223498 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.223479 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.229049 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.229021 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-64206840-openshift-default-dockercfg-pf25x\"" Jul 24 15:25:25.243487 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.243461 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv"] Jul 24 15:25:25.270466 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.270435 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-socket\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.270466 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.270470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-envoy\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.270680 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.270491 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740204f8-7248-4c27-99b3-f6809f8311d4-istiod-ca-cert\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.270680 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.270546 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-credential-socket\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.270680 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.270589 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hns8d\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-kube-api-access-hns8d\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.270680 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.270631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740204f8-7248-4c27-99b3-f6809f8311d4-istio-podinfo\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.270825 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.270688 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-certs\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.270825 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.270732 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-istio-token\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.270825 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.270752 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-data\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.371605 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.371566 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-certs\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.371817 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.371617 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-istio-token\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.371817 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.371636 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-data\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.371817 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.371682 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-socket\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.371817 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.371710 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-envoy\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.371817 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.371731 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740204f8-7248-4c27-99b3-f6809f8311d4-istiod-ca-cert\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.371817 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.371760 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-credential-socket\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.371817 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.371793 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hns8d\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-kube-api-access-hns8d\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.372168 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.371869 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740204f8-7248-4c27-99b3-f6809f8311d4-istio-podinfo\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.372369 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.372342 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-certs\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.372474 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.372379 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-socket\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.372474 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.372456 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-credential-socket\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.372474 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.372468 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-data\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.372721 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.372702 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740204f8-7248-4c27-99b3-f6809f8311d4-istiod-ca-cert\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.374422 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.374404 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-envoy\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.374743 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.374723 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740204f8-7248-4c27-99b3-f6809f8311d4-istio-podinfo\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.381075 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.381051 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-istio-token\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.381208 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.381165 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hns8d\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-kube-api-access-hns8d\") pod \"e2e-derive-64206840-openshift-default-5db67bd896-5vnlv\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.533495 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.533387 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:25.686040 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.686009 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv"] Jul 24 15:25:25.689927 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:25:25.689897 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740204f8_7248_4c27_99b3_f6809f8311d4.slice/crio-874e35c3119163154029e2f5c1b13477765fbdebaad0ebcd707be295039b697d WatchSource:0}: Error finding container 874e35c3119163154029e2f5c1b13477765fbdebaad0ebcd707be295039b697d: Status 404 returned error can't find the container with id 874e35c3119163154029e2f5c1b13477765fbdebaad0ebcd707be295039b697d Jul 24 15:25:25.692232 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.692175 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:25:25.692336 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.692284 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:25:25.692336 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.692327 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:25:25.984091 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.984056 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" event={"ID":"740204f8-7248-4c27-99b3-f6809f8311d4","Type":"ContainerStarted","Data":"e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027"} Jul 24 15:25:25.984342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:25.984114 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" event={"ID":"740204f8-7248-4c27-99b3-f6809f8311d4","Type":"ContainerStarted","Data":"874e35c3119163154029e2f5c1b13477765fbdebaad0ebcd707be295039b697d"} Jul 24 15:25:26.009702 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:26.009642 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" podStartSLOduration=1.009616894 podStartE2EDuration="1.009616894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:25:26.009618122 +0000 UTC m=+1744.957111710" watchObservedRunningTime="2026-07-24 15:25:26.009616894 +0000 UTC m=+1744.957110481" Jul 24 15:25:26.534448 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:26.534409 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:26.539155 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:26.539127 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:26.988911 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:26.988881 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:26.989864 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:26.989845 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:27.448673 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.448634 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf"] Jul 24 15:25:27.452225 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.452204 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.455779 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.455756 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-64206840-serving-cert\"" Jul 24 15:25:27.455887 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.455817 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n8r5x\"" Jul 24 15:25:27.469548 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.469515 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf"] Jul 24 15:25:27.489080 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.489046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-openshift-service-ca\") pod \"maas-api-e2e-derive-64206840-564478754c-kdqkf\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.489258 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.489087 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-maas-api-tls\") pod \"maas-api-e2e-derive-64206840-564478754c-kdqkf\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.489258 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.489133 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqct9\" (UniqueName: \"kubernetes.io/projected/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-kube-api-access-kqct9\") pod \"maas-api-e2e-derive-64206840-564478754c-kdqkf\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.589940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.589892 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-openshift-service-ca\") pod \"maas-api-e2e-derive-64206840-564478754c-kdqkf\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.590436 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.589962 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-maas-api-tls\") pod \"maas-api-e2e-derive-64206840-564478754c-kdqkf\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.590436 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.590013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqct9\" (UniqueName: \"kubernetes.io/projected/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-kube-api-access-kqct9\") pod \"maas-api-e2e-derive-64206840-564478754c-kdqkf\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.590707 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.590682 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-openshift-service-ca\") pod \"maas-api-e2e-derive-64206840-564478754c-kdqkf\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.592786 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.592754 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-maas-api-tls\") pod \"maas-api-e2e-derive-64206840-564478754c-kdqkf\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.600516 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.600484 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqct9\" (UniqueName: \"kubernetes.io/projected/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-kube-api-access-kqct9\") pod \"maas-api-e2e-derive-64206840-564478754c-kdqkf\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.763475 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.763387 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:27.897807 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.897782 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf"] Jul 24 15:25:27.899729 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:25:27.899703 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b7d72d_1e18_4608_bcb1_cca8bc5a57d5.slice/crio-a6e75280f36a8050f62f5bfde711215f5b130c2785b9a77dcb20eb765bd1380d WatchSource:0}: Error finding container a6e75280f36a8050f62f5bfde711215f5b130c2785b9a77dcb20eb765bd1380d: Status 404 returned error can't find the container with id a6e75280f36a8050f62f5bfde711215f5b130c2785b9a77dcb20eb765bd1380d Jul 24 15:25:27.994366 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:27.994329 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" event={"ID":"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5","Type":"ContainerStarted","Data":"a6e75280f36a8050f62f5bfde711215f5b130c2785b9a77dcb20eb765bd1380d"} Jul 24 15:25:29.000962 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.000923 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" event={"ID":"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5","Type":"ContainerStarted","Data":"e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521"} Jul 24 15:25:29.022581 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.022529 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" podStartSLOduration=1.570887982 podStartE2EDuration="2.022513453s" totalImagesPullingTime="451.625471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:25:29.02179906 +0000 UTC m=+1747.969292670" watchObservedRunningTime="2026-07-24 15:25:29.022513453 +0000 UTC m=+1747.970007039" Jul 24 15:25:29.697138 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.697075 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:25:29.811431 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811398 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-envoy\") pod \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " Jul 24 15:25:29.811612 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811443 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-data\") pod \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " Jul 24 15:25:29.811612 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811471 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istiod-ca-cert\") pod \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " Jul 24 15:25:29.811612 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811509 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-token\") pod \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " Jul 24 15:25:29.811612 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811583 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-socket\") pod \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " Jul 24 15:25:29.811846 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811616 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-podinfo\") pod \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " Jul 24 15:25:29.811846 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811646 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52kmv\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-kube-api-access-52kmv\") pod \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " Jul 24 15:25:29.811846 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811681 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-certs\") pod \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " Jul 24 15:25:29.811846 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811752 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-credential-socket\") pod \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\" (UID: \"e50f44fa-663b-4149-b73a-c9c7dd0664bd\") " Jul 24 15:25:29.812039 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811895 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e50f44fa-663b-4149-b73a-c9c7dd0664bd" (UID: "e50f44fa-663b-4149-b73a-c9c7dd0664bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:25:29.812039 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.811943 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e50f44fa-663b-4149-b73a-c9c7dd0664bd" (UID: "e50f44fa-663b-4149-b73a-c9c7dd0664bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:25:29.812152 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.812118 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e50f44fa-663b-4149-b73a-c9c7dd0664bd" (UID: "e50f44fa-663b-4149-b73a-c9c7dd0664bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:25:29.812248 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.812229 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:29.812307 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.812254 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:29.812342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.812321 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e50f44fa-663b-4149-b73a-c9c7dd0664bd" (UID: "e50f44fa-663b-4149-b73a-c9c7dd0664bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:25:29.812342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.812306 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e50f44fa-663b-4149-b73a-c9c7dd0664bd" (UID: "e50f44fa-663b-4149-b73a-c9c7dd0664bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:25:29.813891 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.813863 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e50f44fa-663b-4149-b73a-c9c7dd0664bd" (UID: "e50f44fa-663b-4149-b73a-c9c7dd0664bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:25:29.813993 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.813968 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-kube-api-access-52kmv" (OuterVolumeSpecName: "kube-api-access-52kmv") pod "e50f44fa-663b-4149-b73a-c9c7dd0664bd" (UID: "e50f44fa-663b-4149-b73a-c9c7dd0664bd"). InnerVolumeSpecName "kube-api-access-52kmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:25:29.814060 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.814030 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e50f44fa-663b-4149-b73a-c9c7dd0664bd" (UID: "e50f44fa-663b-4149-b73a-c9c7dd0664bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:25:29.814109 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.814095 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e50f44fa-663b-4149-b73a-c9c7dd0664bd" (UID: "e50f44fa-663b-4149-b73a-c9c7dd0664bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:25:29.913790 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.913752 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:29.913790 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.913779 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:29.913790 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.913789 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:29.913790 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.913797 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:29.914059 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.913805 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50f44fa-663b-4149-b73a-c9c7dd0664bd-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:29.914059 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.913815 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52kmv\" (UniqueName: \"kubernetes.io/projected/e50f44fa-663b-4149-b73a-c9c7dd0664bd-kube-api-access-52kmv\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:29.914059 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:29.913826 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50f44fa-663b-4149-b73a-c9c7dd0664bd-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:30.007005 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.006921 2557 generic.go:356] "Generic (PLEG): container finished" podID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerID="b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75" exitCode=0 Jul 24 15:25:30.007005 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.006972 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" Jul 24 15:25:30.007529 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.007023 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" event={"ID":"e50f44fa-663b-4149-b73a-c9c7dd0664bd","Type":"ContainerDied","Data":"b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75"} Jul 24 15:25:30.007529 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.007079 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg" event={"ID":"e50f44fa-663b-4149-b73a-c9c7dd0664bd","Type":"ContainerDied","Data":"9244e56765193ae5fbc38b45328faa41f2dbfd013939b0c4b0ca67e7f4b753e1"} Jul 24 15:25:30.007529 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.007112 2557 scope.go:117] "RemoveContainer" containerID="b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75" Jul 24 15:25:30.007529 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.007314 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:30.016518 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.016498 2557 scope.go:117] "RemoveContainer" containerID="54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926" Jul 24 15:25:30.024644 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.024624 2557 scope.go:117] "RemoveContainer" containerID="b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75" Jul 24 15:25:30.024929 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:25:30.024909 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75\": container with ID starting with b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75 not found: ID does not exist" containerID="b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75" Jul 24 15:25:30.024994 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.024940 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75"} err="failed to get container status \"b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75\": rpc error: code = NotFound desc = could not find container \"b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75\": container with ID starting with b77e5f46aa20d0e2ae70184989b97302bb1bf8145dd56448d1c49a5cc2f7ef75 not found: ID does not exist" Jul 24 15:25:30.024994 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.024960 2557 scope.go:117] "RemoveContainer" containerID="54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926" Jul 24 15:25:30.025226 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:25:30.025209 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926\": container with ID starting with 54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926 not found: ID does not exist" containerID="54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926" Jul 24 15:25:30.025279 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.025233 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926"} err="failed to get container status \"54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926\": rpc error: code = NotFound desc = could not find container \"54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926\": container with ID starting with 54fc51650f5fa650f60c726d70319960b6dd5125e17bf36e3137d63148b26926 not found: ID does not exist" Jul 24 15:25:30.031493 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.031463 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg"] Jul 24 15:25:30.041771 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:30.041750 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e2f4813e-openshift-default-647c58f8f6-xdvwg"] Jul 24 15:25:31.588634 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:31.588602 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" path="/var/lib/kubelet/pods/e50f44fa-663b-4149-b73a-c9c7dd0664bd/volumes" Jul 24 15:25:36.019033 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:36.019000 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:25:41.070264 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:41.070213 2557 generic.go:356] "Generic (PLEG): container finished" podID="740204f8-7248-4c27-99b3-f6809f8311d4" containerID="e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027" exitCode=0 Jul 24 15:25:41.070728 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:41.070224 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" event={"ID":"740204f8-7248-4c27-99b3-f6809f8311d4","Type":"ContainerDied","Data":"e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027"} Jul 24 15:25:41.070801 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:41.070788 2557 scope.go:117] "RemoveContainer" containerID="e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027" Jul 24 15:25:41.071786 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:41.071727 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:25:41.071883 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:41.071787 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:25:41.071883 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:41.071815 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:25:41.989266 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:41.989220 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:42.077586 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:42.077553 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" event={"ID":"740204f8-7248-4c27-99b3-f6809f8311d4","Type":"ContainerStarted","Data":"bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6"} Jul 24 15:25:42.534135 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:42.534094 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:42.535241 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:42.535218 2557 patch_prober.go:28] interesting pod/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:25:42.535366 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:42.535264 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:25:43.533960 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:43.533920 2557 patch_prober.go:28] interesting pod/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:25:43.534372 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:43.533996 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:25:44.534086 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:44.534048 2557 patch_prober.go:28] interesting pod/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:25:44.534502 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:44.534116 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:25:45.534298 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:45.534257 2557 patch_prober.go:28] interesting pod/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:25:45.534804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:45.534337 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:25:46.538279 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:46.538247 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:46.538758 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:46.538632 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:46.539356 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:46.539337 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:25:52.263996 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.263963 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4"] Jul 24 15:25:52.264413 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.264329 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" Jul 24 15:25:52.264413 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.264341 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" Jul 24 15:25:52.264413 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.264411 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" Jul 24 15:25:52.267418 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.267401 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" Jul 24 15:25:52.270689 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.270667 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:25:52.277570 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.277545 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4"] Jul 24 15:25:52.420577 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.420540 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239ac20-1bac-447c-b266-50621c6cfdd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-64206840-nbwh4\" (UID: \"c239ac20-1bac-447c-b266-50621c6cfdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" Jul 24 15:25:52.521877 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.521783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239ac20-1bac-447c-b266-50621c6cfdd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-64206840-nbwh4\" (UID: \"c239ac20-1bac-447c-b266-50621c6cfdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" Jul 24 15:25:52.522463 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.522445 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239ac20-1bac-447c-b266-50621c6cfdd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-64206840-nbwh4\" (UID: \"c239ac20-1bac-447c-b266-50621c6cfdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" Jul 24 15:25:52.577938 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.577895 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" Jul 24 15:25:52.714860 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:52.714830 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4"] Jul 24 15:25:52.716877 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:25:52.716849 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc239ac20_1bac_447c_b266_50621c6cfdd0.slice/crio-adb48ffd78380ca3fe96682d8954102a7e2bce466b1f4ffa8ef28c7a29fb0cf9 WatchSource:0}: Error finding container adb48ffd78380ca3fe96682d8954102a7e2bce466b1f4ffa8ef28c7a29fb0cf9: Status 404 returned error can't find the container with id adb48ffd78380ca3fe96682d8954102a7e2bce466b1f4ffa8ef28c7a29fb0cf9 Jul 24 15:25:53.133670 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:53.133632 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" event={"ID":"c239ac20-1bac-447c-b266-50621c6cfdd0","Type":"ContainerStarted","Data":"520702bd999f6e0eaaca8e421fffed4001a3054f1e09d4f50836b11d7b240f88"} Jul 24 15:25:53.133835 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:53.133687 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" event={"ID":"c239ac20-1bac-447c-b266-50621c6cfdd0","Type":"ContainerStarted","Data":"adb48ffd78380ca3fe96682d8954102a7e2bce466b1f4ffa8ef28c7a29fb0cf9"} Jul 24 15:25:53.152114 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:53.152054 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" podStartSLOduration=1.152037166 podStartE2EDuration="1.152037166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:25:53.151574978 +0000 UTC m=+1772.099068565" watchObservedRunningTime="2026-07-24 15:25:53.152037166 +0000 UTC m=+1772.099530753" Jul 24 15:25:54.140020 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:54.139985 2557 generic.go:356] "Generic (PLEG): container finished" podID="c239ac20-1bac-447c-b266-50621c6cfdd0" containerID="520702bd999f6e0eaaca8e421fffed4001a3054f1e09d4f50836b11d7b240f88" exitCode=0 Jul 24 15:25:54.140426 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:54.140070 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" event={"ID":"c239ac20-1bac-447c-b266-50621c6cfdd0","Type":"ContainerDied","Data":"520702bd999f6e0eaaca8e421fffed4001a3054f1e09d4f50836b11d7b240f88"} Jul 24 15:25:55.271975 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:55.271942 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" Jul 24 15:25:55.447127 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:55.447098 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239ac20-1bac-447c-b266-50621c6cfdd0-maas-api-service-ca\") pod \"c239ac20-1bac-447c-b266-50621c6cfdd0\" (UID: \"c239ac20-1bac-447c-b266-50621c6cfdd0\") " Jul 24 15:25:55.447467 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:55.447445 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c239ac20-1bac-447c-b266-50621c6cfdd0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c239ac20-1bac-447c-b266-50621c6cfdd0" (UID: "c239ac20-1bac-447c-b266-50621c6cfdd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:25:55.548624 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:55.548589 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239ac20-1bac-447c-b266-50621c6cfdd0-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:25:56.150664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:56.150636 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" Jul 24 15:25:56.150664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:56.150662 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4" event={"ID":"c239ac20-1bac-447c-b266-50621c6cfdd0","Type":"ContainerDied","Data":"adb48ffd78380ca3fe96682d8954102a7e2bce466b1f4ffa8ef28c7a29fb0cf9"} Jul 24 15:25:56.150871 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:25:56.150704 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adb48ffd78380ca3fe96682d8954102a7e2bce466b1f4ffa8ef28c7a29fb0cf9" Jul 24 15:26:02.761549 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:02.761506 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf"] Jul 24 15:26:02.762080 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:02.761866 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" podUID="e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5" containerName="maas-api" containerID="cri-o://e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521" gracePeriod=30 Jul 24 15:26:03.019811 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.019732 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:26:03.115931 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.115893 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-maas-api-tls\") pod \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " Jul 24 15:26:03.116104 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.115973 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-openshift-service-ca\") pod \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " Jul 24 15:26:03.116104 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.116045 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqct9\" (UniqueName: \"kubernetes.io/projected/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-kube-api-access-kqct9\") pod \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\" (UID: \"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5\") " Jul 24 15:26:03.116424 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.116392 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5" (UID: "e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:26:03.118198 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.118153 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5" (UID: "e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:26:03.118285 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.118263 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-kube-api-access-kqct9" (OuterVolumeSpecName: "kube-api-access-kqct9") pod "e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5" (UID: "e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5"). InnerVolumeSpecName "kube-api-access-kqct9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:26:03.183814 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.183778 2557 generic.go:356] "Generic (PLEG): container finished" podID="e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5" containerID="e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521" exitCode=0 Jul 24 15:26:03.183983 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.183824 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" Jul 24 15:26:03.183983 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.183868 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" event={"ID":"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5","Type":"ContainerDied","Data":"e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521"} Jul 24 15:26:03.183983 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.183921 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf" event={"ID":"e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5","Type":"ContainerDied","Data":"a6e75280f36a8050f62f5bfde711215f5b130c2785b9a77dcb20eb765bd1380d"} Jul 24 15:26:03.183983 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.183945 2557 scope.go:117] "RemoveContainer" containerID="e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521" Jul 24 15:26:03.193530 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.193509 2557 scope.go:117] "RemoveContainer" containerID="e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521" Jul 24 15:26:03.193860 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:26:03.193838 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521\": container with ID starting with e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521 not found: ID does not exist" containerID="e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521" Jul 24 15:26:03.193945 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.193868 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521"} err="failed to get container status \"e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521\": rpc error: code = NotFound desc = could not find container \"e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521\": container with ID starting with e1917ca16b69fd0e9c2cbd3dab4c9d9e2fe0682ba5c3850965ee7c150d20b521 not found: ID does not exist" Jul 24 15:26:03.205481 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.205453 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf"] Jul 24 15:26:03.209584 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.209558 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64206840-564478754c-kdqkf"] Jul 24 15:26:03.217313 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.217289 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqct9\" (UniqueName: \"kubernetes.io/projected/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-kube-api-access-kqct9\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:03.217375 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.217319 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:03.217375 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.217330 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:03.587924 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:03.587892 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5" path="/var/lib/kubelet/pods/e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5/volumes" Jul 24 15:26:07.364030 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:07.363993 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4"] Jul 24 15:26:07.368760 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:07.368736 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64206840-nbwh4"] Jul 24 15:26:07.497398 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:07.497368 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv"] Jul 24 15:26:07.497614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:07.497589 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" containerID="cri-o://bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6" gracePeriod=30 Jul 24 15:26:07.589071 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:07.589028 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c239ac20-1bac-447c-b266-50621c6cfdd0" path="/var/lib/kubelet/pods/c239ac20-1bac-447c-b266-50621c6cfdd0/volumes" Jul 24 15:26:12.756048 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.756022 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:26:12.799123 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799043 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-socket\") pod \"740204f8-7248-4c27-99b3-f6809f8311d4\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " Jul 24 15:26:12.799123 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799090 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740204f8-7248-4c27-99b3-f6809f8311d4-istio-podinfo\") pod \"740204f8-7248-4c27-99b3-f6809f8311d4\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " Jul 24 15:26:12.799123 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799113 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-certs\") pod \"740204f8-7248-4c27-99b3-f6809f8311d4\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " Jul 24 15:26:12.799425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799139 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-envoy\") pod \"740204f8-7248-4c27-99b3-f6809f8311d4\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " Jul 24 15:26:12.799425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799167 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-credential-socket\") pod \"740204f8-7248-4c27-99b3-f6809f8311d4\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " Jul 24 15:26:12.799425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799251 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740204f8-7248-4c27-99b3-f6809f8311d4-istiod-ca-cert\") pod \"740204f8-7248-4c27-99b3-f6809f8311d4\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " Jul 24 15:26:12.799425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799283 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hns8d\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-kube-api-access-hns8d\") pod \"740204f8-7248-4c27-99b3-f6809f8311d4\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " Jul 24 15:26:12.799425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799307 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-data\") pod \"740204f8-7248-4c27-99b3-f6809f8311d4\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " Jul 24 15:26:12.799425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799335 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-istio-token\") pod \"740204f8-7248-4c27-99b3-f6809f8311d4\" (UID: \"740204f8-7248-4c27-99b3-f6809f8311d4\") " Jul 24 15:26:12.799425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799363 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "740204f8-7248-4c27-99b3-f6809f8311d4" (UID: "740204f8-7248-4c27-99b3-f6809f8311d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:26:12.799778 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799565 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:12.799778 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799699 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "740204f8-7248-4c27-99b3-f6809f8311d4" (UID: "740204f8-7248-4c27-99b3-f6809f8311d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:26:12.799778 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799769 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "740204f8-7248-4c27-99b3-f6809f8311d4" (UID: "740204f8-7248-4c27-99b3-f6809f8311d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:26:12.799950 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.799859 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740204f8-7248-4c27-99b3-f6809f8311d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "740204f8-7248-4c27-99b3-f6809f8311d4" (UID: "740204f8-7248-4c27-99b3-f6809f8311d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:26:12.800393 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.800354 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "740204f8-7248-4c27-99b3-f6809f8311d4" (UID: "740204f8-7248-4c27-99b3-f6809f8311d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:26:12.801613 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.801587 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "740204f8-7248-4c27-99b3-f6809f8311d4" (UID: "740204f8-7248-4c27-99b3-f6809f8311d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:26:12.801804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.801787 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "740204f8-7248-4c27-99b3-f6809f8311d4" (UID: "740204f8-7248-4c27-99b3-f6809f8311d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:26:12.801971 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.801952 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/740204f8-7248-4c27-99b3-f6809f8311d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "740204f8-7248-4c27-99b3-f6809f8311d4" (UID: "740204f8-7248-4c27-99b3-f6809f8311d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:26:12.802047 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.802029 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-kube-api-access-hns8d" (OuterVolumeSpecName: "kube-api-access-hns8d") pod "740204f8-7248-4c27-99b3-f6809f8311d4" (UID: "740204f8-7248-4c27-99b3-f6809f8311d4"). InnerVolumeSpecName "kube-api-access-hns8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:26:12.900531 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.900501 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:12.900531 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.900529 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740204f8-7248-4c27-99b3-f6809f8311d4-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:12.900531 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.900539 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hns8d\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-kube-api-access-hns8d\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:12.900754 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.900549 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:12.900754 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.900558 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740204f8-7248-4c27-99b3-f6809f8311d4-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:12.900754 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.900566 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740204f8-7248-4c27-99b3-f6809f8311d4-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:12.900754 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.900573 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:12.900754 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:12.900581 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740204f8-7248-4c27-99b3-f6809f8311d4-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:26:13.232098 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.232059 2557 generic.go:356] "Generic (PLEG): container finished" podID="740204f8-7248-4c27-99b3-f6809f8311d4" containerID="bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6" exitCode=0 Jul 24 15:26:13.232337 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.232109 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" Jul 24 15:26:13.232337 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.232129 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" event={"ID":"740204f8-7248-4c27-99b3-f6809f8311d4","Type":"ContainerDied","Data":"bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6"} Jul 24 15:26:13.232337 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.232210 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv" event={"ID":"740204f8-7248-4c27-99b3-f6809f8311d4","Type":"ContainerDied","Data":"874e35c3119163154029e2f5c1b13477765fbdebaad0ebcd707be295039b697d"} Jul 24 15:26:13.232337 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.232243 2557 scope.go:117] "RemoveContainer" containerID="bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6" Jul 24 15:26:13.241310 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.241269 2557 scope.go:117] "RemoveContainer" containerID="e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027" Jul 24 15:26:13.249277 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.249259 2557 scope.go:117] "RemoveContainer" containerID="bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6" Jul 24 15:26:13.249554 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:26:13.249534 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6\": container with ID starting with bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6 not found: ID does not exist" containerID="bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6" Jul 24 15:26:13.249601 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.249564 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6"} err="failed to get container status \"bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6\": rpc error: code = NotFound desc = could not find container \"bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6\": container with ID starting with bb106f062218f93cfc5ca9cc7857fd95f45295dec82296f448329b4963a37ef6 not found: ID does not exist" Jul 24 15:26:13.249601 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.249581 2557 scope.go:117] "RemoveContainer" containerID="e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027" Jul 24 15:26:13.249788 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:26:13.249774 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027\": container with ID starting with e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027 not found: ID does not exist" containerID="e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027" Jul 24 15:26:13.249828 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.249791 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027"} err="failed to get container status \"e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027\": rpc error: code = NotFound desc = could not find container \"e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027\": container with ID starting with e0a8d279d6f081d3083973ea76d33c7b0a8a6e9c916f740eeec33556a2f2d027 not found: ID does not exist" Jul 24 15:26:13.255135 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.255115 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv"] Jul 24 15:26:13.260992 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.260969 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-64206840-openshift-default-5db67bd896-5vnlv"] Jul 24 15:26:13.587839 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:26:13.587765 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" path="/var/lib/kubelet/pods/740204f8-7248-4c27-99b3-f6809f8311d4/volumes" Jul 24 15:27:47.445243 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445141 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm"] Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445462 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445474 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445493 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c239ac20-1bac-447c-b266-50621c6cfdd0" containerName="revoke-keys" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445499 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c239ac20-1bac-447c-b266-50621c6cfdd0" containerName="revoke-keys" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445506 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5" containerName="maas-api" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445511 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5" containerName="maas-api" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445524 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445529 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445535 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445540 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445582 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445590 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b7d72d-1e18-4608-bcb1-cca8bc5a57d5" containerName="maas-api" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445596 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c239ac20-1bac-447c-b266-50621c6cfdd0" containerName="revoke-keys" Jul 24 15:27:47.445664 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.445603 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e50f44fa-663b-4149-b73a-c9c7dd0664bd" containerName="istio-proxy" Jul 24 15:27:47.448391 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.448376 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.453800 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.453774 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-61f9c21e-openshift-default-dockercfg-ffdqx\"" Jul 24 15:27:47.483094 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.483062 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm"] Jul 24 15:27:47.608385 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.608349 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-data\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.608569 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.608389 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-socket\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.608569 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.608415 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istiod-ca-cert\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.608569 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.608441 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-certs\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.608569 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.608479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-podinfo\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.608569 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.608543 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvkj\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-kube-api-access-msvkj\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.608787 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.608601 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-token\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.608787 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.608639 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-credential-socket\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.608787 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.608663 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-envoy\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709339 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709248 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-data\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709339 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709287 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-socket\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709557 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709411 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istiod-ca-cert\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709557 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709442 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-certs\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709557 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709476 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-podinfo\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709557 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709519 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvkj\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-kube-api-access-msvkj\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709557 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709542 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-token\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709582 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-credential-socket\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-envoy\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709658 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-socket\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709703 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-data\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.709954 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.709916 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-credential-socket\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.710058 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.710039 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-certs\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.710333 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.710317 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istiod-ca-cert\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.711942 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.711923 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-envoy\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.712152 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.712135 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-podinfo\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.717982 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.717961 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvkj\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-kube-api-access-msvkj\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.718141 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.718120 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-token\") pod \"e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.757874 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.757844 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:47.894523 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.894488 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm"] Jul 24 15:27:47.896964 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:27:47.896929 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f391a5_8c4f_4abd_a6a9_3ce96179ddc5.slice/crio-2cee58cf19ba841054fc779f8901b741d56efc4eb6f8626660afd35d46d11035 WatchSource:0}: Error finding container 2cee58cf19ba841054fc779f8901b741d56efc4eb6f8626660afd35d46d11035: Status 404 returned error can't find the container with id 2cee58cf19ba841054fc779f8901b741d56efc4eb6f8626660afd35d46d11035 Jul 24 15:27:47.899054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.899025 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:27:47.899149 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.899098 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:27:47.899149 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:47.899138 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:27:48.661282 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:48.661243 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" event={"ID":"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5","Type":"ContainerStarted","Data":"48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7"} Jul 24 15:27:48.661678 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:48.661302 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" event={"ID":"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5","Type":"ContainerStarted","Data":"2cee58cf19ba841054fc779f8901b741d56efc4eb6f8626660afd35d46d11035"} Jul 24 15:27:48.758570 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:48.758535 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:48.763490 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:48.763466 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:48.786367 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:48.786316 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" podStartSLOduration=1.78629959 podStartE2EDuration="1.78629959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:27:48.684648424 +0000 UTC m=+1887.632142023" watchObservedRunningTime="2026-07-24 15:27:48.78629959 +0000 UTC m=+1887.733793176" Jul 24 15:27:49.665765 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:49.665736 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:49.666673 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:49.666654 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:27:50.896160 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:50.896124 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77"] Jul 24 15:27:50.896738 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:50.896718 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="740204f8-7248-4c27-99b3-f6809f8311d4" containerName="istio-proxy" Jul 24 15:27:50.899623 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:50.899592 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:50.904383 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:50.904294 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-61f9c21e-serving-cert\"" Jul 24 15:27:50.904567 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:50.904545 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n8r5x\"" Jul 24 15:27:50.915946 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:50.915924 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77"] Jul 24 15:27:51.042001 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.041966 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15059838-65c8-4012-8f9a-bf287d18bea8-maas-api-tls\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.042163 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.042083 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjz5d\" (UniqueName: \"kubernetes.io/projected/15059838-65c8-4012-8f9a-bf287d18bea8-kube-api-access-vjz5d\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.042163 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.042139 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15059838-65c8-4012-8f9a-bf287d18bea8-openshift-service-ca\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.143027 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.142992 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz5d\" (UniqueName: \"kubernetes.io/projected/15059838-65c8-4012-8f9a-bf287d18bea8-kube-api-access-vjz5d\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.143215 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.143044 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15059838-65c8-4012-8f9a-bf287d18bea8-openshift-service-ca\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.143215 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.143092 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15059838-65c8-4012-8f9a-bf287d18bea8-maas-api-tls\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.143316 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:27:51.143240 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-serving-cert: secret "maas-api-e2e-mt-61f9c21e-serving-cert" not found Jul 24 15:27:51.143316 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:27:51.143308 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15059838-65c8-4012-8f9a-bf287d18bea8-maas-api-tls podName:15059838-65c8-4012-8f9a-bf287d18bea8 nodeName:}" failed. No retries permitted until 2026-07-24 15:27:51.64328842 +0000 UTC m=+1890.590781994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15059838-65c8-4012-8f9a-bf287d18bea8-maas-api-tls") pod "maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" (UID: "15059838-65c8-4012-8f9a-bf287d18bea8") : secret "maas-api-e2e-mt-61f9c21e-serving-cert" not found Jul 24 15:27:51.143799 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.143780 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15059838-65c8-4012-8f9a-bf287d18bea8-openshift-service-ca\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.156361 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.156341 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjz5d\" (UniqueName: \"kubernetes.io/projected/15059838-65c8-4012-8f9a-bf287d18bea8-kube-api-access-vjz5d\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.648896 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.648813 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15059838-65c8-4012-8f9a-bf287d18bea8-maas-api-tls\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.651268 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.651240 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15059838-65c8-4012-8f9a-bf287d18bea8-maas-api-tls\") pod \"maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.811132 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.811096 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:51.965033 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:51.965006 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77"] Jul 24 15:27:51.968129 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:27:51.968095 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15059838_65c8_4012_8f9a_bf287d18bea8.slice/crio-f90f77b4bb442d485d025f2bfd13ab3631864675f4ecdfda787c18a9715554d9 WatchSource:0}: Error finding container f90f77b4bb442d485d025f2bfd13ab3631864675f4ecdfda787c18a9715554d9: Status 404 returned error can't find the container with id f90f77b4bb442d485d025f2bfd13ab3631864675f4ecdfda787c18a9715554d9 Jul 24 15:27:52.680164 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:52.680128 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" event={"ID":"15059838-65c8-4012-8f9a-bf287d18bea8","Type":"ContainerStarted","Data":"f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03"} Jul 24 15:27:52.680359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:52.680177 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" event={"ID":"15059838-65c8-4012-8f9a-bf287d18bea8","Type":"ContainerStarted","Data":"f90f77b4bb442d485d025f2bfd13ab3631864675f4ecdfda787c18a9715554d9"} Jul 24 15:27:52.680429 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:52.680357 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:27:52.699978 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:52.699883 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" podStartSLOduration=2.228419549 podStartE2EDuration="2.699867589s" totalImagesPullingTime="471.44804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:27:52.697681255 +0000 UTC m=+1891.645174840" watchObservedRunningTime="2026-07-24 15:27:52.699867589 +0000 UTC m=+1891.647361174" Jul 24 15:27:58.690690 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:27:58.690656 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:28:02.797007 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:02.796967 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs"] Jul 24 15:28:02.807548 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:02.807525 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs"] Jul 24 15:28:02.807682 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:02.807652 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" Jul 24 15:28:02.810201 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:02.810164 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:28:02.852328 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:02.852296 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fcebaa-971b-406b-95aa-72cbf81a3c93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs\" (UID: \"98fcebaa-971b-406b-95aa-72cbf81a3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" Jul 24 15:28:02.953633 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:02.953604 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fcebaa-971b-406b-95aa-72cbf81a3c93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs\" (UID: \"98fcebaa-971b-406b-95aa-72cbf81a3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" Jul 24 15:28:02.954152 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:02.954134 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fcebaa-971b-406b-95aa-72cbf81a3c93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs\" (UID: \"98fcebaa-971b-406b-95aa-72cbf81a3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" Jul 24 15:28:03.119342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.119240 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" Jul 24 15:28:03.253641 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.253468 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs"] Jul 24 15:28:03.256544 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:28:03.256506 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fcebaa_971b_406b_95aa_72cbf81a3c93.slice/crio-414f46692f89ea3ff0615fecca3319427a7c33faaacf5f99eeca38b0ce5d6a2f WatchSource:0}: Error finding container 414f46692f89ea3ff0615fecca3319427a7c33faaacf5f99eeca38b0ce5d6a2f: Status 404 returned error can't find the container with id 414f46692f89ea3ff0615fecca3319427a7c33faaacf5f99eeca38b0ce5d6a2f Jul 24 15:28:03.741425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.741384 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" event={"ID":"98fcebaa-971b-406b-95aa-72cbf81a3c93","Type":"ContainerStarted","Data":"44f80be966570be4ca0809b2f1d80be247be92938fc2b39bdad6533899745279"} Jul 24 15:28:03.741627 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.741458 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" event={"ID":"98fcebaa-971b-406b-95aa-72cbf81a3c93","Type":"ContainerStarted","Data":"414f46692f89ea3ff0615fecca3319427a7c33faaacf5f99eeca38b0ce5d6a2f"} Jul 24 15:28:03.743109 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.743082 2557 generic.go:356] "Generic (PLEG): container finished" podID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerID="48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7" exitCode=0 Jul 24 15:28:03.743259 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.743127 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" event={"ID":"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5","Type":"ContainerDied","Data":"48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7"} Jul 24 15:28:03.743493 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.743479 2557 scope.go:117] "RemoveContainer" containerID="48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7" Jul 24 15:28:03.744469 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.744443 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:28:03.744575 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.744508 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:28:03.744575 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.744539 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:28:03.758344 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:03.758305 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" podStartSLOduration=1.7582908659999998 podStartE2EDuration="1.758290866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:28:03.757003364 +0000 UTC m=+1902.704496951" watchObservedRunningTime="2026-07-24 15:28:03.758290866 +0000 UTC m=+1902.705784452" Jul 24 15:28:04.666565 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:04.666531 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:28:04.749261 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:04.749230 2557 generic.go:356] "Generic (PLEG): container finished" podID="98fcebaa-971b-406b-95aa-72cbf81a3c93" containerID="44f80be966570be4ca0809b2f1d80be247be92938fc2b39bdad6533899745279" exitCode=0 Jul 24 15:28:04.749455 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:04.749262 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" event={"ID":"98fcebaa-971b-406b-95aa-72cbf81a3c93","Type":"ContainerDied","Data":"44f80be966570be4ca0809b2f1d80be247be92938fc2b39bdad6533899745279"} Jul 24 15:28:04.751248 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:04.751226 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" event={"ID":"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5","Type":"ContainerStarted","Data":"64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907"} Jul 24 15:28:04.758812 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:04.758791 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:28:04.759819 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:04.759796 2557 patch_prober.go:28] interesting pod/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 24 15:28:04.759931 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:04.759846 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 24 15:28:05.758903 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:05.758864 2557 patch_prober.go:28] interesting pod/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 24 15:28:05.759222 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:05.758927 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 24 15:28:05.923199 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:05.923166 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" Jul 24 15:28:05.980434 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:05.980404 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fcebaa-971b-406b-95aa-72cbf81a3c93-maas-api-service-ca\") pod \"98fcebaa-971b-406b-95aa-72cbf81a3c93\" (UID: \"98fcebaa-971b-406b-95aa-72cbf81a3c93\") " Jul 24 15:28:05.980721 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:05.980701 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fcebaa-971b-406b-95aa-72cbf81a3c93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98fcebaa-971b-406b-95aa-72cbf81a3c93" (UID: "98fcebaa-971b-406b-95aa-72cbf81a3c93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:28:06.081776 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:06.081693 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98fcebaa-971b-406b-95aa-72cbf81a3c93-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:06.758789 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:06.758758 2557 patch_prober.go:28] interesting pod/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 24 15:28:06.758969 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:06.758812 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 24 15:28:06.761737 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:06.761719 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" Jul 24 15:28:06.761847 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:06.761755 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs" event={"ID":"98fcebaa-971b-406b-95aa-72cbf81a3c93","Type":"ContainerDied","Data":"414f46692f89ea3ff0615fecca3319427a7c33faaacf5f99eeca38b0ce5d6a2f"} Jul 24 15:28:06.761847 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:06.761804 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="414f46692f89ea3ff0615fecca3319427a7c33faaacf5f99eeca38b0ce5d6a2f" Jul 24 15:28:07.758854 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:07.758810 2557 patch_prober.go:28] interesting pod/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 24 15:28:07.759047 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:07.758869 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 24 15:28:08.762854 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:08.762824 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:28:08.763241 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:08.763089 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:28:08.763764 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:08.763742 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:28:12.912275 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:12.912240 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77"] Jul 24 15:28:12.912848 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:12.912495 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" podUID="15059838-65c8-4012-8f9a-bf287d18bea8" containerName="maas-api" containerID="cri-o://f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03" gracePeriod=30 Jul 24 15:28:13.173625 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.173555 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:28:13.266038 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.266000 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15059838-65c8-4012-8f9a-bf287d18bea8-maas-api-tls\") pod \"15059838-65c8-4012-8f9a-bf287d18bea8\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " Jul 24 15:28:13.266272 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.266064 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15059838-65c8-4012-8f9a-bf287d18bea8-openshift-service-ca\") pod \"15059838-65c8-4012-8f9a-bf287d18bea8\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " Jul 24 15:28:13.266272 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.266108 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjz5d\" (UniqueName: \"kubernetes.io/projected/15059838-65c8-4012-8f9a-bf287d18bea8-kube-api-access-vjz5d\") pod \"15059838-65c8-4012-8f9a-bf287d18bea8\" (UID: \"15059838-65c8-4012-8f9a-bf287d18bea8\") " Jul 24 15:28:13.266465 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.266439 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15059838-65c8-4012-8f9a-bf287d18bea8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15059838-65c8-4012-8f9a-bf287d18bea8" (UID: "15059838-65c8-4012-8f9a-bf287d18bea8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:28:13.268425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.268394 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15059838-65c8-4012-8f9a-bf287d18bea8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15059838-65c8-4012-8f9a-bf287d18bea8" (UID: "15059838-65c8-4012-8f9a-bf287d18bea8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:28:13.268425 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.268394 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15059838-65c8-4012-8f9a-bf287d18bea8-kube-api-access-vjz5d" (OuterVolumeSpecName: "kube-api-access-vjz5d") pod "15059838-65c8-4012-8f9a-bf287d18bea8" (UID: "15059838-65c8-4012-8f9a-bf287d18bea8"). InnerVolumeSpecName "kube-api-access-vjz5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:28:13.366940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.366904 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15059838-65c8-4012-8f9a-bf287d18bea8-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:13.366940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.366935 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjz5d\" (UniqueName: \"kubernetes.io/projected/15059838-65c8-4012-8f9a-bf287d18bea8-kube-api-access-vjz5d\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:13.366940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.366945 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15059838-65c8-4012-8f9a-bf287d18bea8-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:13.794799 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.794753 2557 generic.go:356] "Generic (PLEG): container finished" podID="15059838-65c8-4012-8f9a-bf287d18bea8" containerID="f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03" exitCode=0 Jul 24 15:28:13.794976 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.794812 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" Jul 24 15:28:13.794976 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.794844 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" event={"ID":"15059838-65c8-4012-8f9a-bf287d18bea8","Type":"ContainerDied","Data":"f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03"} Jul 24 15:28:13.794976 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.794938 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77" event={"ID":"15059838-65c8-4012-8f9a-bf287d18bea8","Type":"ContainerDied","Data":"f90f77b4bb442d485d025f2bfd13ab3631864675f4ecdfda787c18a9715554d9"} Jul 24 15:28:13.794976 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.794968 2557 scope.go:117] "RemoveContainer" containerID="f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03" Jul 24 15:28:13.803543 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.803529 2557 scope.go:117] "RemoveContainer" containerID="f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03" Jul 24 15:28:13.803819 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:28:13.803797 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03\": container with ID starting with f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03 not found: ID does not exist" containerID="f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03" Jul 24 15:28:13.803919 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.803825 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03"} err="failed to get container status \"f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03\": rpc error: code = NotFound desc = could not find container \"f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03\": container with ID starting with f44e03b8746d78c791b40d0cccd0ebf6f9004d98d3bdc90558700ce31856ac03 not found: ID does not exist" Jul 24 15:28:13.812112 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.812078 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77"] Jul 24 15:28:13.813716 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:13.813700 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61f9c21e-6fd4d69484-j6p77"] Jul 24 15:28:15.587467 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:15.587434 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15059838-65c8-4012-8f9a-bf287d18bea8" path="/var/lib/kubelet/pods/15059838-65c8-4012-8f9a-bf287d18bea8/volumes" Jul 24 15:28:17.880204 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:17.880142 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs"] Jul 24 15:28:17.882012 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:17.881990 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61f9c21e-8vhxs"] Jul 24 15:28:19.587161 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:19.587129 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98fcebaa-971b-406b-95aa-72cbf81a3c93" path="/var/lib/kubelet/pods/98fcebaa-971b-406b-95aa-72cbf81a3c93/volumes" Jul 24 15:28:24.511603 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:24.511564 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm"] Jul 24 15:28:24.512045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:24.511789 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" containerID="cri-o://64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907" gracePeriod=30 Jul 24 15:28:29.760338 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.760308 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:28:29.872932 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.872844 2557 generic.go:356] "Generic (PLEG): container finished" podID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerID="64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907" exitCode=0 Jul 24 15:28:29.872932 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.872893 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" Jul 24 15:28:29.872932 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.872901 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" event={"ID":"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5","Type":"ContainerDied","Data":"64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907"} Jul 24 15:28:29.873142 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.872951 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm" event={"ID":"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5","Type":"ContainerDied","Data":"2cee58cf19ba841054fc779f8901b741d56efc4eb6f8626660afd35d46d11035"} Jul 24 15:28:29.873142 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.872978 2557 scope.go:117] "RemoveContainer" containerID="64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907" Jul 24 15:28:29.881402 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.881385 2557 scope.go:117] "RemoveContainer" containerID="48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7" Jul 24 15:28:29.888757 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.888741 2557 scope.go:117] "RemoveContainer" containerID="64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907" Jul 24 15:28:29.889029 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:28:29.889010 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907\": container with ID starting with 64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907 not found: ID does not exist" containerID="64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907" Jul 24 15:28:29.889074 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.889039 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907"} err="failed to get container status \"64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907\": rpc error: code = NotFound desc = could not find container \"64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907\": container with ID starting with 64c4889efdb5489a4545ca77eee16ac56264d6a183d575cd8ca3b5999bf39907 not found: ID does not exist" Jul 24 15:28:29.889074 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.889058 2557 scope.go:117] "RemoveContainer" containerID="48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7" Jul 24 15:28:29.889310 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:28:29.889287 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7\": container with ID starting with 48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7 not found: ID does not exist" containerID="48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7" Jul 24 15:28:29.889365 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.889317 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7"} err="failed to get container status \"48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7\": rpc error: code = NotFound desc = could not find container \"48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7\": container with ID starting with 48df8a049f4ade6a8b22657e45941ad4db8bdf9e1e66b69e398eedafb3a28bb7 not found: ID does not exist" Jul 24 15:28:29.910655 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.910632 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-token\") pod \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " Jul 24 15:28:29.910752 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.910668 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istiod-ca-cert\") pod \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " Jul 24 15:28:29.910752 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.910712 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-credential-socket\") pod \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " Jul 24 15:28:29.910843 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.910825 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msvkj\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-kube-api-access-msvkj\") pod \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " Jul 24 15:28:29.910903 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.910889 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-data\") pod \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " Jul 24 15:28:29.910954 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.910917 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-envoy\") pod \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " Jul 24 15:28:29.911000 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.910962 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-socket\") pod \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " Jul 24 15:28:29.911000 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.910966 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" (UID: "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:28:29.911103 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.911021 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-podinfo\") pod \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " Jul 24 15:28:29.911103 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.911047 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-certs\") pod \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\" (UID: \"46f391a5-8c4f-4abd-a6a9-3ce96179ddc5\") " Jul 24 15:28:29.911103 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.911047 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" (UID: "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:28:29.911291 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.911265 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" (UID: "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:28:29.911508 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.911401 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" (UID: "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:28:29.911508 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.911424 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:29.911508 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.911443 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:29.911508 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.911461 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:29.911759 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.911688 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-data" (OuterVolumeSpecName: "istio-data") pod "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" (UID: "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:28:29.913053 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.913034 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-kube-api-access-msvkj" (OuterVolumeSpecName: "kube-api-access-msvkj") pod "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" (UID: "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5"). InnerVolumeSpecName "kube-api-access-msvkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:28:29.913259 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.913238 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" (UID: "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:28:29.913580 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.913558 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-token" (OuterVolumeSpecName: "istio-token") pod "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" (UID: "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:28:29.913627 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:29.913573 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" (UID: "46f391a5-8c4f-4abd-a6a9-3ce96179ddc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:28:30.012804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:30.012758 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvkj\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-kube-api-access-msvkj\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:30.012804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:30.012797 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:30.012804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:30.012806 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:30.012804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:30.012814 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:30.012804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:30.012822 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:30.013095 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:30.012830 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:28:30.195377 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:30.195353 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm"] Jul 24 15:28:30.199750 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:30.199726 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-61f9c21e-openshift-default-7bb6c5d64-sjskm"] Jul 24 15:28:31.588577 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:31.588544 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" path="/var/lib/kubelet/pods/46f391a5-8c4f-4abd-a6a9-3ce96179ddc5/volumes" Jul 24 15:28:44.712322 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712266 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972"] Jul 24 15:28:44.712751 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712736 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" Jul 24 15:28:44.712807 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712754 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" Jul 24 15:28:44.712807 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712770 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fcebaa-971b-406b-95aa-72cbf81a3c93" containerName="revoke-keys" Jul 24 15:28:44.712807 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712778 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fcebaa-971b-406b-95aa-72cbf81a3c93" containerName="revoke-keys" Jul 24 15:28:44.712807 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712794 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15059838-65c8-4012-8f9a-bf287d18bea8" containerName="maas-api" Jul 24 15:28:44.712807 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712804 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="15059838-65c8-4012-8f9a-bf287d18bea8" containerName="maas-api" Jul 24 15:28:44.712957 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712897 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="98fcebaa-971b-406b-95aa-72cbf81a3c93" containerName="revoke-keys" Jul 24 15:28:44.712957 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712910 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" Jul 24 15:28:44.712957 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712920 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="15059838-65c8-4012-8f9a-bf287d18bea8" containerName="maas-api" Jul 24 15:28:44.712957 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.712928 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" Jul 24 15:28:44.716309 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.716283 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.719031 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.719013 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5ad3cd4c-openshift-default-dockercfg-kqxr2\"" Jul 24 15:28:44.727361 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.727338 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972"] Jul 24 15:28:44.740169 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.740141 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-certs\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.740311 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.740206 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-credential-socket\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.740311 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.740235 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-istio-token\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.740311 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.740293 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgtzh\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-kube-api-access-jgtzh\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.740524 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.740355 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfa0398-1114-48db-8fdf-b182a037570e-istiod-ca-cert\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.740524 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.740397 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfa0398-1114-48db-8fdf-b182a037570e-istio-podinfo\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.740524 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.740446 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-socket\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.740524 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.740474 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-data\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.740524 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.740507 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-envoy\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.841632 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.841598 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfa0398-1114-48db-8fdf-b182a037570e-istiod-ca-cert\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.841823 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.841646 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfa0398-1114-48db-8fdf-b182a037570e-istio-podinfo\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.841823 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.841685 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-socket\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.841823 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.841706 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-data\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.841823 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.841729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-envoy\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.841823 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.841790 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-certs\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.842089 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.841843 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-credential-socket\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.842089 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.841868 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-istio-token\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.842089 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.841901 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgtzh\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-kube-api-access-jgtzh\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.842274 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.842255 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-socket\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.842335 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.842296 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-data\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.842509 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.842489 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfa0398-1114-48db-8fdf-b182a037570e-istiod-ca-cert\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.842605 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.842513 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-certs\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.842652 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.842636 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-credential-socket\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.844705 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.844684 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-envoy\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.844785 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.844739 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfa0398-1114-48db-8fdf-b182a037570e-istio-podinfo\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.850907 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.850882 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-istio-token\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:44.851172 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:44.851151 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgtzh\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-kube-api-access-jgtzh\") pod \"e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:45.026481 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:45.026388 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:45.158367 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:45.158344 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972"] Jul 24 15:28:45.160078 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:28:45.160053 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfa0398_1114_48db_8fdf_b182a037570e.slice/crio-a49a2f8bec5ff1b0a8749383e5c5701c52b31db54fac1b7c26a66b24c2916c06 WatchSource:0}: Error finding container a49a2f8bec5ff1b0a8749383e5c5701c52b31db54fac1b7c26a66b24c2916c06: Status 404 returned error can't find the container with id a49a2f8bec5ff1b0a8749383e5c5701c52b31db54fac1b7c26a66b24c2916c06 Jul 24 15:28:45.162266 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:45.162231 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:28:45.162369 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:45.162300 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:28:45.162369 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:45.162329 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:28:45.945621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:45.945580 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" event={"ID":"bdfa0398-1114-48db-8fdf-b182a037570e","Type":"ContainerStarted","Data":"ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83"} Jul 24 15:28:45.946010 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:45.945639 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" event={"ID":"bdfa0398-1114-48db-8fdf-b182a037570e","Type":"ContainerStarted","Data":"a49a2f8bec5ff1b0a8749383e5c5701c52b31db54fac1b7c26a66b24c2916c06"} Jul 24 15:28:45.966747 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:45.966699 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" podStartSLOduration=1.966685228 podStartE2EDuration="1.966685228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:28:45.965577161 +0000 UTC m=+1944.913070748" watchObservedRunningTime="2026-07-24 15:28:45.966685228 +0000 UTC m=+1944.914178815" Jul 24 15:28:46.026865 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:46.026829 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:46.031291 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:46.031264 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:46.950266 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:46.950223 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:46.951147 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:46.951127 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:28:48.057758 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.057719 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x"] Jul 24 15:28:48.058234 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.058217 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" Jul 24 15:28:48.058288 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.058238 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f391a5-8c4f-4abd-a6a9-3ce96179ddc5" containerName="istio-proxy" Jul 24 15:28:48.061428 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.061411 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.066760 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.066739 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5ad3cd4c-serving-cert\"" Jul 24 15:28:48.066760 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.066754 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n8r5x\"" Jul 24 15:28:48.072797 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.072770 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x"] Jul 24 15:28:48.168062 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.168021 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hfp\" (UniqueName: \"kubernetes.io/projected/20cefa46-e598-428d-a79b-eb79cb6e69c9-kube-api-access-h8hfp\") pod \"maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.168255 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.168074 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20cefa46-e598-428d-a79b-eb79cb6e69c9-maas-api-tls\") pod \"maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.168255 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.168097 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cefa46-e598-428d-a79b-eb79cb6e69c9-openshift-service-ca\") pod \"maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.268916 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.268881 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20cefa46-e598-428d-a79b-eb79cb6e69c9-maas-api-tls\") pod \"maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.268916 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.268921 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cefa46-e598-428d-a79b-eb79cb6e69c9-openshift-service-ca\") pod \"maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.269165 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.269013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hfp\" (UniqueName: \"kubernetes.io/projected/20cefa46-e598-428d-a79b-eb79cb6e69c9-kube-api-access-h8hfp\") pod \"maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.269603 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.269583 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cefa46-e598-428d-a79b-eb79cb6e69c9-openshift-service-ca\") pod \"maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.271430 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.271415 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20cefa46-e598-428d-a79b-eb79cb6e69c9-maas-api-tls\") pod \"maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.283745 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.283717 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hfp\" (UniqueName: \"kubernetes.io/projected/20cefa46-e598-428d-a79b-eb79cb6e69c9-kube-api-access-h8hfp\") pod \"maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.372042 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.371948 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:48.510707 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.510683 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x"] Jul 24 15:28:48.512865 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:28:48.512832 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cefa46_e598_428d_a79b_eb79cb6e69c9.slice/crio-a47d547e3f735556c80c732f451a2f3a9610499d774719882c3192b877e73f24 WatchSource:0}: Error finding container a47d547e3f735556c80c732f451a2f3a9610499d774719882c3192b877e73f24: Status 404 returned error can't find the container with id a47d547e3f735556c80c732f451a2f3a9610499d774719882c3192b877e73f24 Jul 24 15:28:48.514319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.514302 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:28:48.960435 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:48.960395 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" event={"ID":"20cefa46-e598-428d-a79b-eb79cb6e69c9","Type":"ContainerStarted","Data":"a47d547e3f735556c80c732f451a2f3a9610499d774719882c3192b877e73f24"} Jul 24 15:28:49.966354 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:49.966316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" event={"ID":"20cefa46-e598-428d-a79b-eb79cb6e69c9","Type":"ContainerStarted","Data":"cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245"} Jul 24 15:28:49.966817 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:49.966403 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:28:49.987933 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:49.987885 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" podStartSLOduration=1.53528238 podStartE2EDuration="1.987870573s" totalImagesPullingTime="452.588193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:28:49.985596407 +0000 UTC m=+1948.933090005" watchObservedRunningTime="2026-07-24 15:28:49.987870573 +0000 UTC m=+1948.935364158" Jul 24 15:28:55.977974 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:28:55.977945 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:29:02.943208 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:02.942941 2557 patch_prober.go:28] interesting pod/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:43722->10.134.0.42:15021: read: connection reset by peer" start-of-body= Jul 24 15:29:02.943208 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:02.943008 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:43722->10.134.0.42:15021: read: connection reset by peer" Jul 24 15:29:03.035734 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:03.035704 2557 generic.go:356] "Generic (PLEG): container finished" podID="bdfa0398-1114-48db-8fdf-b182a037570e" containerID="ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83" exitCode=0 Jul 24 15:29:03.035894 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:03.035749 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" event={"ID":"bdfa0398-1114-48db-8fdf-b182a037570e","Type":"ContainerDied","Data":"ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83"} Jul 24 15:29:03.036108 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:03.036096 2557 scope.go:117] "RemoveContainer" containerID="ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83" Jul 24 15:29:03.037101 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:03.037074 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:29:03.037155 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:03.037142 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:29:03.037210 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:03.037171 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:29:04.042704 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:04.042669 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" event={"ID":"bdfa0398-1114-48db-8fdf-b182a037570e","Type":"ContainerStarted","Data":"9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe"} Jul 24 15:29:05.027274 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:05.027235 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:29:05.027442 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:05.027370 2557 patch_prober.go:28] interesting pod/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 24 15:29:05.027442 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:05.027416 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 24 15:29:06.026979 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:06.026941 2557 patch_prober.go:28] interesting pod/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 24 15:29:06.027422 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:06.026999 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 24 15:29:07.027590 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:07.027551 2557 patch_prober.go:28] interesting pod/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 24 15:29:07.027978 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:07.027620 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 24 15:29:08.030013 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:08.029983 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:29:08.030405 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:08.030215 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:29:08.031023 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:08.031005 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:29:21.887267 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:21.887169 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j"] Jul 24 15:29:21.890110 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:21.890090 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" Jul 24 15:29:21.892746 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:21.892718 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:29:21.898100 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:21.898074 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j"] Jul 24 15:29:22.069374 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:22.069337 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j\" (UID: \"0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" Jul 24 15:29:22.170363 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:22.170328 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j\" (UID: \"0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" Jul 24 15:29:22.170935 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:22.170914 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j\" (UID: \"0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" Jul 24 15:29:22.202284 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:22.202259 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" Jul 24 15:29:22.547847 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:22.545838 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j"] Jul 24 15:29:22.552244 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:29:22.552213 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0519ce1e_fc9d_4b9d_b895_f0b6c3eb6a87.slice/crio-3546b5833cf907e376900bdd80f665bbf4a444a3c054170a821744d5bf67c497 WatchSource:0}: Error finding container 3546b5833cf907e376900bdd80f665bbf4a444a3c054170a821744d5bf67c497: Status 404 returned error can't find the container with id 3546b5833cf907e376900bdd80f665bbf4a444a3c054170a821744d5bf67c497 Jul 24 15:29:23.131892 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:23.131856 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" event={"ID":"0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87","Type":"ContainerStarted","Data":"328f01606800e6481a873ce1f63f2c752271561d03ed7628ef1405153cc24d32"} Jul 24 15:29:23.132357 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:23.131912 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" event={"ID":"0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87","Type":"ContainerStarted","Data":"3546b5833cf907e376900bdd80f665bbf4a444a3c054170a821744d5bf67c497"} Jul 24 15:29:23.149162 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:23.149104 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" podStartSLOduration=2.149082948 podStartE2EDuration="2.149082948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:29:23.148339923 +0000 UTC m=+1982.095833509" watchObservedRunningTime="2026-07-24 15:29:23.149082948 +0000 UTC m=+1982.096576535" Jul 24 15:29:24.138207 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:24.138145 2557 generic.go:356] "Generic (PLEG): container finished" podID="0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87" containerID="328f01606800e6481a873ce1f63f2c752271561d03ed7628ef1405153cc24d32" exitCode=0 Jul 24 15:29:24.138207 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:24.138207 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" event={"ID":"0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87","Type":"ContainerDied","Data":"328f01606800e6481a873ce1f63f2c752271561d03ed7628ef1405153cc24d32"} Jul 24 15:29:25.272293 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:25.272271 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" Jul 24 15:29:25.401633 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:25.401541 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87-maas-api-service-ca\") pod \"0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87\" (UID: \"0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87\") " Jul 24 15:29:25.401915 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:25.401894 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87" (UID: "0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:29:25.502848 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:25.502819 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:29:26.149004 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:26.148967 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" event={"ID":"0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87","Type":"ContainerDied","Data":"3546b5833cf907e376900bdd80f665bbf4a444a3c054170a821744d5bf67c497"} Jul 24 15:29:26.149178 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:26.149010 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j" Jul 24 15:29:26.149178 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:26.149018 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3546b5833cf907e376900bdd80f665bbf4a444a3c054170a821744d5bf67c497" Jul 24 15:29:37.490723 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.490692 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x"] Jul 24 15:29:37.491170 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.490922 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" podUID="20cefa46-e598-428d-a79b-eb79cb6e69c9" containerName="maas-api" containerID="cri-o://cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245" gracePeriod=30 Jul 24 15:29:37.747272 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.747170 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:29:37.810144 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.810109 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8hfp\" (UniqueName: \"kubernetes.io/projected/20cefa46-e598-428d-a79b-eb79cb6e69c9-kube-api-access-h8hfp\") pod \"20cefa46-e598-428d-a79b-eb79cb6e69c9\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " Jul 24 15:29:37.810339 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.810156 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20cefa46-e598-428d-a79b-eb79cb6e69c9-maas-api-tls\") pod \"20cefa46-e598-428d-a79b-eb79cb6e69c9\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " Jul 24 15:29:37.810339 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.810195 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cefa46-e598-428d-a79b-eb79cb6e69c9-openshift-service-ca\") pod \"20cefa46-e598-428d-a79b-eb79cb6e69c9\" (UID: \"20cefa46-e598-428d-a79b-eb79cb6e69c9\") " Jul 24 15:29:37.810609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.810581 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20cefa46-e598-428d-a79b-eb79cb6e69c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20cefa46-e598-428d-a79b-eb79cb6e69c9" (UID: "20cefa46-e598-428d-a79b-eb79cb6e69c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:29:37.812308 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.812285 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cefa46-e598-428d-a79b-eb79cb6e69c9-kube-api-access-h8hfp" (OuterVolumeSpecName: "kube-api-access-h8hfp") pod "20cefa46-e598-428d-a79b-eb79cb6e69c9" (UID: "20cefa46-e598-428d-a79b-eb79cb6e69c9"). InnerVolumeSpecName "kube-api-access-h8hfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:29:37.812475 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.812453 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20cefa46-e598-428d-a79b-eb79cb6e69c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20cefa46-e598-428d-a79b-eb79cb6e69c9" (UID: "20cefa46-e598-428d-a79b-eb79cb6e69c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:29:37.911029 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.910992 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8hfp\" (UniqueName: \"kubernetes.io/projected/20cefa46-e598-428d-a79b-eb79cb6e69c9-kube-api-access-h8hfp\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:29:37.911029 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.911021 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20cefa46-e598-428d-a79b-eb79cb6e69c9-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:29:37.911029 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:37.911032 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cefa46-e598-428d-a79b-eb79cb6e69c9-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:29:38.212213 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:38.212158 2557 generic.go:356] "Generic (PLEG): container finished" podID="20cefa46-e598-428d-a79b-eb79cb6e69c9" containerID="cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245" exitCode=0 Jul 24 15:29:38.212379 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:38.212225 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" event={"ID":"20cefa46-e598-428d-a79b-eb79cb6e69c9","Type":"ContainerDied","Data":"cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245"} Jul 24 15:29:38.212379 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:38.212284 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" event={"ID":"20cefa46-e598-428d-a79b-eb79cb6e69c9","Type":"ContainerDied","Data":"a47d547e3f735556c80c732f451a2f3a9610499d774719882c3192b877e73f24"} Jul 24 15:29:38.212379 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:38.212308 2557 scope.go:117] "RemoveContainer" containerID="cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245" Jul 24 15:29:38.212379 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:38.212237 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x" Jul 24 15:29:38.221659 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:38.221643 2557 scope.go:117] "RemoveContainer" containerID="cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245" Jul 24 15:29:38.221959 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:29:38.221923 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245\": container with ID starting with cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245 not found: ID does not exist" containerID="cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245" Jul 24 15:29:38.222051 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:38.221958 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245"} err="failed to get container status \"cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245\": rpc error: code = NotFound desc = could not find container \"cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245\": container with ID starting with cf37db8f9b9c0ecdfe692503ccacb2e5c85073c87f8c8426e0833315d5296245 not found: ID does not exist" Jul 24 15:29:38.234556 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:38.234525 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x"] Jul 24 15:29:38.238720 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:38.238695 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ad3cd4c-6bf68c8894-42b2x"] Jul 24 15:29:39.587037 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:39.587006 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20cefa46-e598-428d-a79b-eb79cb6e69c9" path="/var/lib/kubelet/pods/20cefa46-e598-428d-a79b-eb79cb6e69c9/volumes" Jul 24 15:29:41.980363 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:41.980332 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j"] Jul 24 15:29:41.984629 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:41.984604 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ad3cd4c-pg52j"] Jul 24 15:29:43.587200 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:29:43.587151 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87" path="/var/lib/kubelet/pods/0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87/volumes" Jul 24 15:30:00.148044 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.148009 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp"] Jul 24 15:30:00.148498 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.148359 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20cefa46-e598-428d-a79b-eb79cb6e69c9" containerName="maas-api" Jul 24 15:30:00.148498 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.148371 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="20cefa46-e598-428d-a79b-eb79cb6e69c9" containerName="maas-api" Jul 24 15:30:00.148498 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.148392 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87" containerName="revoke-keys" Jul 24 15:30:00.148498 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.148398 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87" containerName="revoke-keys" Jul 24 15:30:00.148498 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.148455 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="20cefa46-e598-428d-a79b-eb79cb6e69c9" containerName="maas-api" Jul 24 15:30:00.148498 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.148467 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0519ce1e-fc9d-4b9d-b895-f0b6c3eb6a87" containerName="revoke-keys" Jul 24 15:30:00.150427 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.150411 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp" Jul 24 15:30:00.153353 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.153335 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:30:00.158718 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.158694 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp"] Jul 24 15:30:00.160881 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.160858 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp" Jul 24 15:30:00.305222 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.305176 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp"] Jul 24 15:30:00.307316 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:30:00.307290 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349676d2_73cd_4a7c_935c_3be4bb5d9593.slice/crio-760eaddba2cc49b15f137b80b7c3636f2ff43a92f03a026cb41f6731d71ee6c1 WatchSource:0}: Error finding container 760eaddba2cc49b15f137b80b7c3636f2ff43a92f03a026cb41f6731d71ee6c1: Status 404 returned error can't find the container with id 760eaddba2cc49b15f137b80b7c3636f2ff43a92f03a026cb41f6731d71ee6c1 Jul 24 15:30:00.317127 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:00.317099 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp" event={"ID":"349676d2-73cd-4a7c-935c-3be4bb5d9593","Type":"ContainerStarted","Data":"760eaddba2cc49b15f137b80b7c3636f2ff43a92f03a026cb41f6731d71ee6c1"} Jul 24 15:30:01.323521 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:01.323486 2557 generic.go:356] "Generic (PLEG): container finished" podID="349676d2-73cd-4a7c-935c-3be4bb5d9593" containerID="990530145a9615a23009aed382ab2c2f2976f7e0276389d511c68c1ee86a7d90" exitCode=0 Jul 24 15:30:01.323929 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:01.323542 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp" event={"ID":"349676d2-73cd-4a7c-935c-3be4bb5d9593","Type":"ContainerDied","Data":"990530145a9615a23009aed382ab2c2f2976f7e0276389d511c68c1ee86a7d90"} Jul 24 15:30:02.458148 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:02.458123 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp" Jul 24 15:30:03.335233 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:03.335208 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp" Jul 24 15:30:03.335233 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:03.335215 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748450-4m5mp" event={"ID":"349676d2-73cd-4a7c-935c-3be4bb5d9593","Type":"ContainerDied","Data":"760eaddba2cc49b15f137b80b7c3636f2ff43a92f03a026cb41f6731d71ee6c1"} Jul 24 15:30:03.335453 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:03.335265 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760eaddba2cc49b15f137b80b7c3636f2ff43a92f03a026cb41f6731d71ee6c1" Jul 24 15:30:21.640291 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:21.640249 2557 scope.go:117] "RemoveContainer" containerID="da798ffb6f42663cdbf7dbaeddcac2b73d2f51be3f82ff1f029d9caef75ccff6" Jul 24 15:30:39.592775 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:39.592744 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972"] Jul 24 15:30:39.593305 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:39.592941 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" containerID="cri-o://9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe" gracePeriod=30 Jul 24 15:30:44.836343 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.836315 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:30:44.883725 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.883639 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-istio-token\") pod \"bdfa0398-1114-48db-8fdf-b182a037570e\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " Jul 24 15:30:44.883725 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.883671 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-envoy\") pod \"bdfa0398-1114-48db-8fdf-b182a037570e\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " Jul 24 15:30:44.883905 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.883767 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-data\") pod \"bdfa0398-1114-48db-8fdf-b182a037570e\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " Jul 24 15:30:44.883905 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.883808 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-certs\") pod \"bdfa0398-1114-48db-8fdf-b182a037570e\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " Jul 24 15:30:44.883905 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.883828 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-credential-socket\") pod \"bdfa0398-1114-48db-8fdf-b182a037570e\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " Jul 24 15:30:44.883905 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.883862 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-socket\") pod \"bdfa0398-1114-48db-8fdf-b182a037570e\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " Jul 24 15:30:44.883905 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.883886 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfa0398-1114-48db-8fdf-b182a037570e-istiod-ca-cert\") pod \"bdfa0398-1114-48db-8fdf-b182a037570e\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " Jul 24 15:30:44.884152 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.883934 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgtzh\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-kube-api-access-jgtzh\") pod \"bdfa0398-1114-48db-8fdf-b182a037570e\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " Jul 24 15:30:44.884152 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.883959 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfa0398-1114-48db-8fdf-b182a037570e-istio-podinfo\") pod \"bdfa0398-1114-48db-8fdf-b182a037570e\" (UID: \"bdfa0398-1114-48db-8fdf-b182a037570e\") " Jul 24 15:30:44.884152 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.884106 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bdfa0398-1114-48db-8fdf-b182a037570e" (UID: "bdfa0398-1114-48db-8fdf-b182a037570e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:30:44.884152 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.884124 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bdfa0398-1114-48db-8fdf-b182a037570e" (UID: "bdfa0398-1114-48db-8fdf-b182a037570e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:30:44.884381 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.884301 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:30:44.884381 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.884324 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:30:44.884484 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.884351 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bdfa0398-1114-48db-8fdf-b182a037570e" (UID: "bdfa0398-1114-48db-8fdf-b182a037570e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:30:44.884705 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.884649 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-data" (OuterVolumeSpecName: "istio-data") pod "bdfa0398-1114-48db-8fdf-b182a037570e" (UID: "bdfa0398-1114-48db-8fdf-b182a037570e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:30:44.884705 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.884681 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfa0398-1114-48db-8fdf-b182a037570e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bdfa0398-1114-48db-8fdf-b182a037570e" (UID: "bdfa0398-1114-48db-8fdf-b182a037570e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:30:44.885976 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.885952 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-istio-token" (OuterVolumeSpecName: "istio-token") pod "bdfa0398-1114-48db-8fdf-b182a037570e" (UID: "bdfa0398-1114-48db-8fdf-b182a037570e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:30:44.886122 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.886108 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdfa0398-1114-48db-8fdf-b182a037570e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bdfa0398-1114-48db-8fdf-b182a037570e" (UID: "bdfa0398-1114-48db-8fdf-b182a037570e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:30:44.886168 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.886119 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bdfa0398-1114-48db-8fdf-b182a037570e" (UID: "bdfa0398-1114-48db-8fdf-b182a037570e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:30:44.886535 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.886520 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-kube-api-access-jgtzh" (OuterVolumeSpecName: "kube-api-access-jgtzh") pod "bdfa0398-1114-48db-8fdf-b182a037570e" (UID: "bdfa0398-1114-48db-8fdf-b182a037570e"). InnerVolumeSpecName "kube-api-access-jgtzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:30:44.985054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.985015 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:30:44.985054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.985044 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:30:44.985054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.985053 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfa0398-1114-48db-8fdf-b182a037570e-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:30:44.985054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.985062 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgtzh\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-kube-api-access-jgtzh\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:30:44.985384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.985070 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfa0398-1114-48db-8fdf-b182a037570e-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:30:44.985384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.985079 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfa0398-1114-48db-8fdf-b182a037570e-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:30:44.985384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:44.985088 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfa0398-1114-48db-8fdf-b182a037570e-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:30:45.538250 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.538215 2557 generic.go:356] "Generic (PLEG): container finished" podID="bdfa0398-1114-48db-8fdf-b182a037570e" containerID="9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe" exitCode=0 Jul 24 15:30:45.538450 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.538260 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" Jul 24 15:30:45.538450 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.538287 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" event={"ID":"bdfa0398-1114-48db-8fdf-b182a037570e","Type":"ContainerDied","Data":"9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe"} Jul 24 15:30:45.538450 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.538330 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972" event={"ID":"bdfa0398-1114-48db-8fdf-b182a037570e","Type":"ContainerDied","Data":"a49a2f8bec5ff1b0a8749383e5c5701c52b31db54fac1b7c26a66b24c2916c06"} Jul 24 15:30:45.538450 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.538352 2557 scope.go:117] "RemoveContainer" containerID="9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe" Jul 24 15:30:45.547343 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.547323 2557 scope.go:117] "RemoveContainer" containerID="ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83" Jul 24 15:30:45.554772 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.554757 2557 scope.go:117] "RemoveContainer" containerID="9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe" Jul 24 15:30:45.555016 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:30:45.554995 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe\": container with ID starting with 9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe not found: ID does not exist" containerID="9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe" Jul 24 15:30:45.555088 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.555025 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe"} err="failed to get container status \"9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe\": rpc error: code = NotFound desc = could not find container \"9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe\": container with ID starting with 9e5a1f17aa8aba9a0c98ec13f59e23a71fb3404bd87ffaa640866907eef4aefe not found: ID does not exist" Jul 24 15:30:45.555088 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.555042 2557 scope.go:117] "RemoveContainer" containerID="ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83" Jul 24 15:30:45.555672 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:30:45.555655 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83\": container with ID starting with ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83 not found: ID does not exist" containerID="ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83" Jul 24 15:30:45.555732 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.555680 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83"} err="failed to get container status \"ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83\": rpc error: code = NotFound desc = could not find container \"ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83\": container with ID starting with ce4265a8deda88341837862a29533f934c171e59687f21b30d9d75490b81ea83 not found: ID does not exist" Jul 24 15:30:45.560439 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.560416 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972"] Jul 24 15:30:45.567342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.567319 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5ad3cd4c-openshift-default-85469d4f9b-fr972"] Jul 24 15:30:45.586718 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:30:45.586698 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" path="/var/lib/kubelet/pods/bdfa0398-1114-48db-8fdf-b182a037570e/volumes" Jul 24 15:31:21.665676 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:21.665638 2557 scope.go:117] "RemoveContainer" containerID="c437400ce8a15c1dd3a1288c7492e4ae55a26ac8b5109b3cb54d0651b866cccf" Jul 24 15:31:21.673687 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:21.673665 2557 scope.go:117] "RemoveContainer" containerID="f8db8f0fdd330595b78f55acfed376a7547e52bec22b66493d2bad3ff8d566f6" Jul 24 15:31:35.918892 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.918862 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg"] Jul 24 15:31:35.919263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.919175 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" Jul 24 15:31:35.919263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.919207 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" Jul 24 15:31:35.919263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.919222 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" Jul 24 15:31:35.919263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.919227 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" Jul 24 15:31:35.919263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.919238 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="349676d2-73cd-4a7c-935c-3be4bb5d9593" containerName="cleanup" Jul 24 15:31:35.919263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.919244 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="349676d2-73cd-4a7c-935c-3be4bb5d9593" containerName="cleanup" Jul 24 15:31:35.919459 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.919291 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" Jul 24 15:31:35.919459 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.919300 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="349676d2-73cd-4a7c-935c-3be4bb5d9593" containerName="cleanup" Jul 24 15:31:35.919459 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.919305 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfa0398-1114-48db-8fdf-b182a037570e" containerName="istio-proxy" Jul 24 15:31:35.922064 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.922049 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:35.924593 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.924570 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-92cb39-openshift-default-dockercfg-55sxt\"" Jul 24 15:31:35.937136 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:35.937109 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg"] Jul 24 15:31:36.015580 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.015543 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-token\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.015773 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.015601 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-data\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.015773 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.015631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-credential-socket\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.015773 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.015668 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-podinfo\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.015773 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.015732 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istiod-ca-cert\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.015978 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.015794 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-envoy\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.015978 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.015835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzt5\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-kube-api-access-7dzt5\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.015978 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.015898 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-certs\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.015978 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.015930 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-socket\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117087 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117052 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-token\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117308 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117095 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-data\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117308 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-credential-socket\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117308 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117156 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-podinfo\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117308 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117177 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istiod-ca-cert\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117308 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117248 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-envoy\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117308 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117279 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzt5\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-kube-api-access-7dzt5\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117636 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117317 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-certs\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117636 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117346 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-socket\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117636 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117559 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-data\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117801 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117721 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-socket\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117801 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117740 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-credential-socket\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.117801 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.117783 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-certs\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.118033 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.118014 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istiod-ca-cert\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.119580 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.119559 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-envoy\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.119826 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.119809 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-podinfo\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.125465 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.125444 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-token\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.125899 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.125881 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzt5\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-kube-api-access-7dzt5\") pod \"e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.231756 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.231658 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:36.365253 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.365224 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg"] Jul 24 15:31:36.368592 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:31:36.368556 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9006acec_0fe5_4aa4_ab00_f9de1aea6a72.slice/crio-4143fb4781630e24df6dc200628d5cde293aa63c2c2d48a09e5a7f053b48044f WatchSource:0}: Error finding container 4143fb4781630e24df6dc200628d5cde293aa63c2c2d48a09e5a7f053b48044f: Status 404 returned error can't find the container with id 4143fb4781630e24df6dc200628d5cde293aa63c2c2d48a09e5a7f053b48044f Jul 24 15:31:36.370806 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.370775 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:36.370907 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.370853 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:36.370907 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.370894 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:36.772792 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.772756 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" event={"ID":"9006acec-0fe5-4aa4-ab00-f9de1aea6a72","Type":"ContainerStarted","Data":"f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623"} Jul 24 15:31:36.772963 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.772809 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" event={"ID":"9006acec-0fe5-4aa4-ab00-f9de1aea6a72","Type":"ContainerStarted","Data":"4143fb4781630e24df6dc200628d5cde293aa63c2c2d48a09e5a7f053b48044f"} Jul 24 15:31:36.793064 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:36.793008 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" podStartSLOduration=1.79299422 podStartE2EDuration="1.79299422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:31:36.791383262 +0000 UTC m=+2115.738876850" watchObservedRunningTime="2026-07-24 15:31:36.79299422 +0000 UTC m=+2115.740487806" Jul 24 15:31:37.231946 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:37.231911 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:37.236371 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:37.236345 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:37.777883 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:37.777852 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:37.778726 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:37.778709 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:38.827351 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:38.827318 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz"] Jul 24 15:31:38.830712 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:38.830695 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:38.833769 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:38.833741 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-92cb39-serving-cert\"" Jul 24 15:31:38.833885 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:38.833753 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n8r5x\"" Jul 24 15:31:38.844051 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:38.844021 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz"] Jul 24 15:31:38.943045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:38.943002 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b222880a-c117-44e0-a807-09744dc26357-openshift-service-ca\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:38.943251 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:38.943147 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxjng\" (UniqueName: \"kubernetes.io/projected/b222880a-c117-44e0-a807-09744dc26357-kube-api-access-mxjng\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:38.943251 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:38.943201 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b222880a-c117-44e0-a807-09744dc26357-maas-api-tls\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:39.044234 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:39.044168 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxjng\" (UniqueName: \"kubernetes.io/projected/b222880a-c117-44e0-a807-09744dc26357-kube-api-access-mxjng\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:39.044392 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:39.044250 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b222880a-c117-44e0-a807-09744dc26357-maas-api-tls\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:39.044392 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:39.044280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b222880a-c117-44e0-a807-09744dc26357-openshift-service-ca\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:39.044484 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:31:39.044391 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-serving-cert: secret "maas-api-e2e-inf-a-92cb39-serving-cert" not found Jul 24 15:31:39.044484 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:31:39.044471 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b222880a-c117-44e0-a807-09744dc26357-maas-api-tls podName:b222880a-c117-44e0-a807-09744dc26357 nodeName:}" failed. No retries permitted until 2026-07-24 15:31:39.544452807 +0000 UTC m=+2118.491946373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b222880a-c117-44e0-a807-09744dc26357-maas-api-tls") pod "maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" (UID: "b222880a-c117-44e0-a807-09744dc26357") : secret "maas-api-e2e-inf-a-92cb39-serving-cert" not found Jul 24 15:31:39.044952 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:39.044935 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b222880a-c117-44e0-a807-09744dc26357-openshift-service-ca\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:39.057876 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:39.057846 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxjng\" (UniqueName: \"kubernetes.io/projected/b222880a-c117-44e0-a807-09744dc26357-kube-api-access-mxjng\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:39.547694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:39.547656 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b222880a-c117-44e0-a807-09744dc26357-maas-api-tls\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:39.550220 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:39.550178 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b222880a-c117-44e0-a807-09744dc26357-maas-api-tls\") pod \"maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:39.741849 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:39.741763 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:39.900122 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:39.900099 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz"] Jul 24 15:31:39.902623 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:31:39.902591 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb222880a_c117_44e0_a807_09744dc26357.slice/crio-8628e376781c9b59ab9585f8249894b7950bb6c4ae5c983ea983239cdec72e8f WatchSource:0}: Error finding container 8628e376781c9b59ab9585f8249894b7950bb6c4ae5c983ea983239cdec72e8f: Status 404 returned error can't find the container with id 8628e376781c9b59ab9585f8249894b7950bb6c4ae5c983ea983239cdec72e8f Jul 24 15:31:40.796860 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:40.796819 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" event={"ID":"b222880a-c117-44e0-a807-09744dc26357","Type":"ContainerStarted","Data":"0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c"} Jul 24 15:31:40.797026 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:40.796882 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" event={"ID":"b222880a-c117-44e0-a807-09744dc26357","Type":"ContainerStarted","Data":"8628e376781c9b59ab9585f8249894b7950bb6c4ae5c983ea983239cdec72e8f"} Jul 24 15:31:40.797078 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:40.797026 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:40.814963 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:40.814910 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" podStartSLOduration=2.134114282 podStartE2EDuration="2.814895231s" totalImagesPullingTime="680.780949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:31:40.814143775 +0000 UTC m=+2119.761637362" watchObservedRunningTime="2026-07-24 15:31:40.814895231 +0000 UTC m=+2119.762388819" Jul 24 15:31:43.885396 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.885357 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g"] Jul 24 15:31:43.889067 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.889046 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:43.891953 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.891927 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dde49b-openshift-default-dockercfg-vp7qv\"" Jul 24 15:31:43.899897 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.899873 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g"] Jul 24 15:31:43.992262 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.992222 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-certs\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:43.992596 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.992558 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-data\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:43.992737 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.992609 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-envoy\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:43.992737 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.992643 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-socket\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:43.992737 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.992672 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-token\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:43.992737 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.992701 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-credential-socket\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:43.992985 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.992753 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-podinfo\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:43.992985 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.992835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istiod-ca-cert\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:43.992985 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:43.992880 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdg2\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-kube-api-access-djdg2\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.093612 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.093578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-podinfo\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.093811 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.093617 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istiod-ca-cert\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.093811 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.093670 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdg2\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-kube-api-access-djdg2\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.093811 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.093718 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-certs\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.093811 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.093741 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-data\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.093811 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.093767 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-envoy\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.093811 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.093788 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-socket\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.093811 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.093809 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-token\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.094218 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.093835 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-credential-socket\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.094218 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.094127 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-data\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.094333 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.094212 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-certs\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.094333 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.094244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-credential-socket\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.094440 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.094336 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-socket\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.094532 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.094439 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istiod-ca-cert\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.096306 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.096287 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-podinfo\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.096386 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.096285 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-envoy\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.102682 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.102659 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-token\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.103195 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.103163 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdg2\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-kube-api-access-djdg2\") pod \"e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.199371 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.199337 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:44.345853 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.345827 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g"] Jul 24 15:31:44.347963 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:31:44.347937 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75eab9d2_e8b3_4bf0_9382_f6357b34797c.slice/crio-3b695bc0a8d9590b83ef5ba377ebf470fc774a4cf693ef23270bb87c952f82dc WatchSource:0}: Error finding container 3b695bc0a8d9590b83ef5ba377ebf470fc774a4cf693ef23270bb87c952f82dc: Status 404 returned error can't find the container with id 3b695bc0a8d9590b83ef5ba377ebf470fc774a4cf693ef23270bb87c952f82dc Jul 24 15:31:44.350097 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.350066 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:44.350219 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.350126 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:44.350219 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.350156 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:44.817110 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.817078 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" event={"ID":"75eab9d2-e8b3-4bf0-9382-f6357b34797c","Type":"ContainerStarted","Data":"7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92"} Jul 24 15:31:44.817299 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.817133 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" event={"ID":"75eab9d2-e8b3-4bf0-9382-f6357b34797c","Type":"ContainerStarted","Data":"3b695bc0a8d9590b83ef5ba377ebf470fc774a4cf693ef23270bb87c952f82dc"} Jul 24 15:31:44.838668 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:44.838616 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" podStartSLOduration=1.838602167 podStartE2EDuration="1.838602167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:31:44.836747143 +0000 UTC m=+2123.784240743" watchObservedRunningTime="2026-07-24 15:31:44.838602167 +0000 UTC m=+2123.786095753" Jul 24 15:31:45.200225 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:45.200170 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:45.205279 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:45.205254 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:45.821724 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:45.821694 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:45.822746 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:45.822723 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:31:46.816419 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:46.816378 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:31:46.919704 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:46.919662 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv"] Jul 24 15:31:46.923443 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:46.923421 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:46.926288 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:46.926266 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dde49b-serving-cert\"" Jul 24 15:31:46.940035 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:46.940007 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv"] Jul 24 15:31:47.029661 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.029626 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bff2ad0-1a33-4a5e-83a5-69040527cde4-maas-api-tls\") pod \"maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.029863 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.029711 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bff2ad0-1a33-4a5e-83a5-69040527cde4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.029863 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.029819 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t46ch\" (UniqueName: \"kubernetes.io/projected/8bff2ad0-1a33-4a5e-83a5-69040527cde4-kube-api-access-t46ch\") pod \"maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.131406 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.131307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t46ch\" (UniqueName: \"kubernetes.io/projected/8bff2ad0-1a33-4a5e-83a5-69040527cde4-kube-api-access-t46ch\") pod \"maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.131406 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.131383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bff2ad0-1a33-4a5e-83a5-69040527cde4-maas-api-tls\") pod \"maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.131627 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.131444 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bff2ad0-1a33-4a5e-83a5-69040527cde4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.132103 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.132085 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bff2ad0-1a33-4a5e-83a5-69040527cde4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.134046 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.134023 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bff2ad0-1a33-4a5e-83a5-69040527cde4-maas-api-tls\") pod \"maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.141131 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.141108 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t46ch\" (UniqueName: \"kubernetes.io/projected/8bff2ad0-1a33-4a5e-83a5-69040527cde4-kube-api-access-t46ch\") pod \"maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.235422 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.235391 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:47.416208 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.410331 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv"] Jul 24 15:31:47.421615 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:31:47.421584 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bff2ad0_1a33_4a5e_83a5_69040527cde4.slice/crio-79741910570e86e7e37aeb549cde152b499eab0ac73fe432fc4fa8d1bbef6997 WatchSource:0}: Error finding container 79741910570e86e7e37aeb549cde152b499eab0ac73fe432fc4fa8d1bbef6997: Status 404 returned error can't find the container with id 79741910570e86e7e37aeb549cde152b499eab0ac73fe432fc4fa8d1bbef6997 Jul 24 15:31:47.833877 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:47.833742 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" event={"ID":"8bff2ad0-1a33-4a5e-83a5-69040527cde4","Type":"ContainerStarted","Data":"79741910570e86e7e37aeb549cde152b499eab0ac73fe432fc4fa8d1bbef6997"} Jul 24 15:31:49.844578 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:49.844540 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" event={"ID":"8bff2ad0-1a33-4a5e-83a5-69040527cde4","Type":"ContainerStarted","Data":"9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a"} Jul 24 15:31:49.845005 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:49.844639 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:49.864415 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:49.864356 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" podStartSLOduration=2.507376866 podStartE2EDuration="3.864340482s" totalImagesPullingTime="1.356963616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:31:49.863574495 +0000 UTC m=+2128.811068084" watchObservedRunningTime="2026-07-24 15:31:49.864340482 +0000 UTC m=+2128.811834106" Jul 24 15:31:52.779556 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:52.779519 2557 patch_prober.go:28] interesting pod/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 15:31:52.779971 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:52.779581 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 15:31:52.860250 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:52.860148 2557 generic.go:356] "Generic (PLEG): container finished" podID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerID="f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623" exitCode=0 Jul 24 15:31:52.860250 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:52.860230 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" event={"ID":"9006acec-0fe5-4aa4-ab00-f9de1aea6a72","Type":"ContainerDied","Data":"f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623"} Jul 24 15:31:52.860658 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:52.860639 2557 scope.go:117] "RemoveContainer" containerID="f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623" Jul 24 15:31:52.861730 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:52.861706 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:52.861800 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:52.861774 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:52.861844 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:52.861809 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:53.866561 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:53.866520 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" event={"ID":"9006acec-0fe5-4aa4-ab00-f9de1aea6a72","Type":"ContainerStarted","Data":"e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051"} Jul 24 15:31:54.235147 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:54.235108 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:54.240717 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:54.240351 2557 patch_prober.go:28] interesting pod/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 15:31:54.240717 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:54.240418 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 15:31:55.232249 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:55.232211 2557 patch_prober.go:28] interesting pod/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 15:31:55.232864 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:55.232831 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 15:31:55.854680 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:55.854654 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:31:56.233163 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:56.233123 2557 patch_prober.go:28] interesting pod/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 15:31:56.233703 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:56.233266 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 15:31:57.232300 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:57.232262 2557 patch_prober.go:28] interesting pod/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 15:31:57.232471 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:57.232319 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 15:31:58.236412 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:58.236379 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:58.236857 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:58.236659 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:58.237321 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:58.237300 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:31:59.898329 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:59.898298 2557 generic.go:356] "Generic (PLEG): container finished" podID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerID="7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92" exitCode=0 Jul 24 15:31:59.898828 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:59.898369 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" event={"ID":"75eab9d2-e8b3-4bf0-9382-f6357b34797c","Type":"ContainerDied","Data":"7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92"} Jul 24 15:31:59.898930 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:59.898910 2557 scope.go:117] "RemoveContainer" containerID="7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92" Jul 24 15:31:59.900178 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:59.900142 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:59.900290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:59.900268 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:31:59.900330 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:31:59.900314 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:32:00.821810 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:00.821768 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:32:00.907890 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:00.907855 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" event={"ID":"75eab9d2-e8b3-4bf0-9382-f6357b34797c","Type":"ContainerStarted","Data":"452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7"} Jul 24 15:32:01.201507 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:01.200200 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:32:01.201507 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:01.201019 2557 patch_prober.go:28] interesting pod/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 15:32:01.201507 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:01.201073 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 15:32:02.200569 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:02.200523 2557 patch_prober.go:28] interesting pod/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 15:32:02.201069 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:02.200595 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 15:32:03.199957 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:03.199918 2557 patch_prober.go:28] interesting pod/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 15:32:03.200661 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:03.200267 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 15:32:04.200266 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:04.200223 2557 patch_prober.go:28] interesting pod/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 15:32:04.200491 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:04.200289 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 15:32:05.203551 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:05.203521 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:32:05.203922 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:05.203874 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:32:05.204592 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:05.204574 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:32:21.697877 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:32:21.697804 2557 scope.go:117] "RemoveContainer" containerID="520702bd999f6e0eaaca8e421fffed4001a3054f1e09d4f50836b11d7b240f88" Jul 24 15:33:18.401695 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.401656 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz"] Jul 24 15:33:18.402244 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.401909 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" podUID="b222880a-c117-44e0-a807-09744dc26357" containerName="maas-api" containerID="cri-o://0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c" gracePeriod=30 Jul 24 15:33:18.658453 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.658428 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:33:18.733432 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.733392 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b222880a-c117-44e0-a807-09744dc26357-openshift-service-ca\") pod \"b222880a-c117-44e0-a807-09744dc26357\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " Jul 24 15:33:18.733625 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.733480 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b222880a-c117-44e0-a807-09744dc26357-maas-api-tls\") pod \"b222880a-c117-44e0-a807-09744dc26357\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " Jul 24 15:33:18.733625 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.733508 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxjng\" (UniqueName: \"kubernetes.io/projected/b222880a-c117-44e0-a807-09744dc26357-kube-api-access-mxjng\") pod \"b222880a-c117-44e0-a807-09744dc26357\" (UID: \"b222880a-c117-44e0-a807-09744dc26357\") " Jul 24 15:33:18.733776 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.733750 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b222880a-c117-44e0-a807-09744dc26357-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b222880a-c117-44e0-a807-09744dc26357" (UID: "b222880a-c117-44e0-a807-09744dc26357"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:33:18.735742 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.735709 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b222880a-c117-44e0-a807-09744dc26357-kube-api-access-mxjng" (OuterVolumeSpecName: "kube-api-access-mxjng") pod "b222880a-c117-44e0-a807-09744dc26357" (UID: "b222880a-c117-44e0-a807-09744dc26357"). InnerVolumeSpecName "kube-api-access-mxjng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:33:18.735853 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.735814 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b222880a-c117-44e0-a807-09744dc26357-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b222880a-c117-44e0-a807-09744dc26357" (UID: "b222880a-c117-44e0-a807-09744dc26357"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:33:18.834713 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.834671 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b222880a-c117-44e0-a807-09744dc26357-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:18.834713 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.834707 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxjng\" (UniqueName: \"kubernetes.io/projected/b222880a-c117-44e0-a807-09744dc26357-kube-api-access-mxjng\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:18.834713 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:18.834718 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b222880a-c117-44e0-a807-09744dc26357-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:19.265674 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.265633 2557 generic.go:356] "Generic (PLEG): container finished" podID="b222880a-c117-44e0-a807-09744dc26357" containerID="0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c" exitCode=0 Jul 24 15:33:19.265869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.265703 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" event={"ID":"b222880a-c117-44e0-a807-09744dc26357","Type":"ContainerDied","Data":"0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c"} Jul 24 15:33:19.265869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.265785 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" event={"ID":"b222880a-c117-44e0-a807-09744dc26357","Type":"ContainerDied","Data":"8628e376781c9b59ab9585f8249894b7950bb6c4ae5c983ea983239cdec72e8f"} Jul 24 15:33:19.265869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.265800 2557 scope.go:117] "RemoveContainer" containerID="0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c" Jul 24 15:33:19.266012 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.265705 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz" Jul 24 15:33:19.275709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.275690 2557 scope.go:117] "RemoveContainer" containerID="0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c" Jul 24 15:33:19.275982 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:33:19.275960 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c\": container with ID starting with 0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c not found: ID does not exist" containerID="0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c" Jul 24 15:33:19.276058 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.275997 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c"} err="failed to get container status \"0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c\": rpc error: code = NotFound desc = could not find container \"0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c\": container with ID starting with 0c038f51811857b00ae7630dc979c2b537024ae02e39574da40429c5ea0f196c not found: ID does not exist" Jul 24 15:33:19.288112 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.288079 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz"] Jul 24 15:33:19.292148 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.292124 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-92cb39-6b6c64c7d7-hnvfz"] Jul 24 15:33:19.587883 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:19.587804 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b222880a-c117-44e0-a807-09744dc26357" path="/var/lib/kubelet/pods/b222880a-c117-44e0-a807-09744dc26357/volumes" Jul 24 15:33:29.747400 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:29.747361 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg"] Jul 24 15:33:29.748018 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:29.747597 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" containerID="cri-o://e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051" gracePeriod=30 Jul 24 15:33:34.997983 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:34.997955 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:33:35.083439 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083349 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-socket\") pod \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " Jul 24 15:33:35.083439 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083391 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istiod-ca-cert\") pod \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " Jul 24 15:33:35.083439 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083412 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-data\") pod \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " Jul 24 15:33:35.083439 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083434 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-token\") pod \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " Jul 24 15:33:35.083777 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083453 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-certs\") pod \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " Jul 24 15:33:35.083777 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083477 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-credential-socket\") pod \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " Jul 24 15:33:35.083777 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083499 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-envoy\") pod \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " Jul 24 15:33:35.083777 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083536 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dzt5\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-kube-api-access-7dzt5\") pod \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " Jul 24 15:33:35.083777 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083571 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-podinfo\") pod \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\" (UID: \"9006acec-0fe5-4aa4-ab00-f9de1aea6a72\") " Jul 24 15:33:35.083777 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083660 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9006acec-0fe5-4aa4-ab00-f9de1aea6a72" (UID: "9006acec-0fe5-4aa4-ab00-f9de1aea6a72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:35.084083 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083799 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9006acec-0fe5-4aa4-ab00-f9de1aea6a72" (UID: "9006acec-0fe5-4aa4-ab00-f9de1aea6a72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:35.084083 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083873 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:35.084083 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083872 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9006acec-0fe5-4aa4-ab00-f9de1aea6a72" (UID: "9006acec-0fe5-4aa4-ab00-f9de1aea6a72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:33:35.084083 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.083891 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:35.084083 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.084067 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9006acec-0fe5-4aa4-ab00-f9de1aea6a72" (UID: "9006acec-0fe5-4aa4-ab00-f9de1aea6a72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:35.084560 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.084532 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-data" (OuterVolumeSpecName: "istio-data") pod "9006acec-0fe5-4aa4-ab00-f9de1aea6a72" (UID: "9006acec-0fe5-4aa4-ab00-f9de1aea6a72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:35.086006 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.085980 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9006acec-0fe5-4aa4-ab00-f9de1aea6a72" (UID: "9006acec-0fe5-4aa4-ab00-f9de1aea6a72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:35.086115 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.086071 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9006acec-0fe5-4aa4-ab00-f9de1aea6a72" (UID: "9006acec-0fe5-4aa4-ab00-f9de1aea6a72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:33:35.086416 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.086304 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-token" (OuterVolumeSpecName: "istio-token") pod "9006acec-0fe5-4aa4-ab00-f9de1aea6a72" (UID: "9006acec-0fe5-4aa4-ab00-f9de1aea6a72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:33:35.086491 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.086418 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-kube-api-access-7dzt5" (OuterVolumeSpecName: "kube-api-access-7dzt5") pod "9006acec-0fe5-4aa4-ab00-f9de1aea6a72" (UID: "9006acec-0fe5-4aa4-ab00-f9de1aea6a72"). InnerVolumeSpecName "kube-api-access-7dzt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:33:35.184832 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.184790 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzt5\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-kube-api-access-7dzt5\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:35.184832 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.184824 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:35.184832 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.184835 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:35.184832 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.184843 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:35.185099 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.184851 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:35.185099 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.184858 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:35.185099 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.184866 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006acec-0fe5-4aa4-ab00-f9de1aea6a72-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:35.340421 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.340327 2557 generic.go:356] "Generic (PLEG): container finished" podID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerID="e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051" exitCode=0 Jul 24 15:33:35.340421 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.340377 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" Jul 24 15:33:35.340421 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.340379 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" event={"ID":"9006acec-0fe5-4aa4-ab00-f9de1aea6a72","Type":"ContainerDied","Data":"e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051"} Jul 24 15:33:35.340656 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.340445 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg" event={"ID":"9006acec-0fe5-4aa4-ab00-f9de1aea6a72","Type":"ContainerDied","Data":"4143fb4781630e24df6dc200628d5cde293aa63c2c2d48a09e5a7f053b48044f"} Jul 24 15:33:35.340656 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.340471 2557 scope.go:117] "RemoveContainer" containerID="e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051" Jul 24 15:33:35.349506 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.349492 2557 scope.go:117] "RemoveContainer" containerID="f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623" Jul 24 15:33:35.357381 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.357362 2557 scope.go:117] "RemoveContainer" containerID="e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051" Jul 24 15:33:35.357621 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:33:35.357602 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051\": container with ID starting with e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051 not found: ID does not exist" containerID="e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051" Jul 24 15:33:35.357667 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.357630 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051"} err="failed to get container status \"e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051\": rpc error: code = NotFound desc = could not find container \"e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051\": container with ID starting with e837e2271a4fd3bd0390c7289f46df8cd5f8883892cad4a98ca596b07c5de051 not found: ID does not exist" Jul 24 15:33:35.357667 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.357651 2557 scope.go:117] "RemoveContainer" containerID="f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623" Jul 24 15:33:35.357898 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:33:35.357877 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623\": container with ID starting with f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623 not found: ID does not exist" containerID="f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623" Jul 24 15:33:35.357951 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.357904 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623"} err="failed to get container status \"f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623\": rpc error: code = NotFound desc = could not find container \"f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623\": container with ID starting with f82cf3c4c8e65b905fc3935e45583abf7daf646b4b167b2ae58aea7e43ab0623 not found: ID does not exist" Jul 24 15:33:35.362429 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.362408 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg"] Jul 24 15:33:35.365689 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.365670 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-92cb39-openshift-default-7c4b897c7d-xjsvg"] Jul 24 15:33:35.588276 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:35.588245 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" path="/var/lib/kubelet/pods/9006acec-0fe5-4aa4-ab00-f9de1aea6a72/volumes" Jul 24 15:33:40.724566 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:40.724528 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv"] Jul 24 15:33:40.725054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:40.724831 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" podUID="8bff2ad0-1a33-4a5e-83a5-69040527cde4" containerName="maas-api" containerID="cri-o://9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a" gracePeriod=30 Jul 24 15:33:40.850320 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:40.850282 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" podUID="8bff2ad0-1a33-4a5e-83a5-69040527cde4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.49:8443/health\": dial tcp 10.134.0.49:8443: connect: connection refused" Jul 24 15:33:41.163955 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.163928 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:33:41.236296 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.236263 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bff2ad0-1a33-4a5e-83a5-69040527cde4-openshift-service-ca\") pod \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " Jul 24 15:33:41.236296 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.236296 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bff2ad0-1a33-4a5e-83a5-69040527cde4-maas-api-tls\") pod \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " Jul 24 15:33:41.236524 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.236346 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t46ch\" (UniqueName: \"kubernetes.io/projected/8bff2ad0-1a33-4a5e-83a5-69040527cde4-kube-api-access-t46ch\") pod \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\" (UID: \"8bff2ad0-1a33-4a5e-83a5-69040527cde4\") " Jul 24 15:33:41.236659 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.236635 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bff2ad0-1a33-4a5e-83a5-69040527cde4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bff2ad0-1a33-4a5e-83a5-69040527cde4" (UID: "8bff2ad0-1a33-4a5e-83a5-69040527cde4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:33:41.238502 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.238481 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bff2ad0-1a33-4a5e-83a5-69040527cde4-kube-api-access-t46ch" (OuterVolumeSpecName: "kube-api-access-t46ch") pod "8bff2ad0-1a33-4a5e-83a5-69040527cde4" (UID: "8bff2ad0-1a33-4a5e-83a5-69040527cde4"). InnerVolumeSpecName "kube-api-access-t46ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:33:41.238579 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.238552 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bff2ad0-1a33-4a5e-83a5-69040527cde4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bff2ad0-1a33-4a5e-83a5-69040527cde4" (UID: "8bff2ad0-1a33-4a5e-83a5-69040527cde4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:33:41.337096 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.337010 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t46ch\" (UniqueName: \"kubernetes.io/projected/8bff2ad0-1a33-4a5e-83a5-69040527cde4-kube-api-access-t46ch\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:41.337096 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.337039 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bff2ad0-1a33-4a5e-83a5-69040527cde4-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:41.337096 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.337050 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bff2ad0-1a33-4a5e-83a5-69040527cde4-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:41.368760 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.368723 2557 generic.go:356] "Generic (PLEG): container finished" podID="8bff2ad0-1a33-4a5e-83a5-69040527cde4" containerID="9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a" exitCode=0 Jul 24 15:33:41.368760 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.368761 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" Jul 24 15:33:41.368936 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.368805 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" event={"ID":"8bff2ad0-1a33-4a5e-83a5-69040527cde4","Type":"ContainerDied","Data":"9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a"} Jul 24 15:33:41.368936 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.368850 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv" event={"ID":"8bff2ad0-1a33-4a5e-83a5-69040527cde4","Type":"ContainerDied","Data":"79741910570e86e7e37aeb549cde152b499eab0ac73fe432fc4fa8d1bbef6997"} Jul 24 15:33:41.368936 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.368876 2557 scope.go:117] "RemoveContainer" containerID="9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a" Jul 24 15:33:41.377948 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.377932 2557 scope.go:117] "RemoveContainer" containerID="9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a" Jul 24 15:33:41.378258 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:33:41.378232 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a\": container with ID starting with 9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a not found: ID does not exist" containerID="9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a" Jul 24 15:33:41.378357 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.378263 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a"} err="failed to get container status \"9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a\": rpc error: code = NotFound desc = could not find container \"9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a\": container with ID starting with 9db2fa72027441bbb91be597db3138e99cd745ee00f6d36cfbd87175bea75d8a not found: ID does not exist" Jul 24 15:33:41.390167 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.390139 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv"] Jul 24 15:33:41.394848 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.394828 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dde49b-5fb4445f6-z9tmv"] Jul 24 15:33:41.588486 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:41.588406 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bff2ad0-1a33-4a5e-83a5-69040527cde4" path="/var/lib/kubelet/pods/8bff2ad0-1a33-4a5e-83a5-69040527cde4/volumes" Jul 24 15:33:51.787092 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:51.787004 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g"] Jul 24 15:33:51.787930 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:51.787894 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" containerID="cri-o://452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7" gracePeriod=30 Jul 24 15:33:52.759085 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759045 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw"] Jul 24 15:33:52.759485 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759466 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" Jul 24 15:33:52.759595 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759490 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" Jul 24 15:33:52.759595 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759502 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bff2ad0-1a33-4a5e-83a5-69040527cde4" containerName="maas-api" Jul 24 15:33:52.759595 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759509 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bff2ad0-1a33-4a5e-83a5-69040527cde4" containerName="maas-api" Jul 24 15:33:52.759595 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759517 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" Jul 24 15:33:52.759595 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759524 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" Jul 24 15:33:52.759595 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759546 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b222880a-c117-44e0-a807-09744dc26357" containerName="maas-api" Jul 24 15:33:52.759595 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759555 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b222880a-c117-44e0-a807-09744dc26357" containerName="maas-api" Jul 24 15:33:52.759940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759629 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bff2ad0-1a33-4a5e-83a5-69040527cde4" containerName="maas-api" Jul 24 15:33:52.759940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759643 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" Jul 24 15:33:52.759940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759655 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9006acec-0fe5-4aa4-ab00-f9de1aea6a72" containerName="istio-proxy" Jul 24 15:33:52.759940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.759663 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b222880a-c117-44e0-a807-09744dc26357" containerName="maas-api" Jul 24 15:33:52.764078 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.764054 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:52.766710 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.766679 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f97fff-openshift-default-dockercfg-m7cjq\"" Jul 24 15:33:52.773978 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.773950 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw"] Jul 24 15:33:52.942009 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.941970 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9791f829-4cd3-46c7-8090-a97d6a630584-istio-podinfo\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:52.942009 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.942006 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-envoy\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:52.942533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.942028 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-certs\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:52.942533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.942149 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-credential-socket\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:52.942533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.942228 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-socket\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:52.942533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.942264 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9791f829-4cd3-46c7-8090-a97d6a630584-istiod-ca-cert\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:52.942533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.942324 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-istio-token\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:52.942533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.942361 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-data\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:52.942533 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:52.942393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctm8\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-kube-api-access-tctm8\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.042961 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.042852 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-credential-socket\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.042961 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.042896 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-socket\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.042961 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.042916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9791f829-4cd3-46c7-8090-a97d6a630584-istiod-ca-cert\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.042961 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.042939 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-istio-token\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.043338 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.042971 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-data\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.043338 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.042995 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctm8\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-kube-api-access-tctm8\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.043338 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.043056 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9791f829-4cd3-46c7-8090-a97d6a630584-istio-podinfo\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.043338 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.043081 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-envoy\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.043338 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.043108 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-certs\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.043634 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.043387 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-credential-socket\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.043634 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.043421 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-socket\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.043634 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.043497 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-certs\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.043634 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.043589 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-data\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.044079 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.044045 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9791f829-4cd3-46c7-8090-a97d6a630584-istiod-ca-cert\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.045558 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.045531 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-envoy\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.045858 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.045838 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9791f829-4cd3-46c7-8090-a97d6a630584-istio-podinfo\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.052091 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.052060 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-istio-token\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.052273 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.052249 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctm8\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-kube-api-access-tctm8\") pod \"e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.075600 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.075570 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:53.215072 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.215032 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw"] Jul 24 15:33:53.217352 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:33:53.217325 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9791f829_4cd3_46c7_8090_a97d6a630584.slice/crio-f54c0a4226f4a5f77da0516c54fc4ad3d10148fed15465776e151a0548425ade WatchSource:0}: Error finding container f54c0a4226f4a5f77da0516c54fc4ad3d10148fed15465776e151a0548425ade: Status 404 returned error can't find the container with id f54c0a4226f4a5f77da0516c54fc4ad3d10148fed15465776e151a0548425ade Jul 24 15:33:53.219395 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.219361 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:33:53.219492 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.219426 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:33:53.219492 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.219459 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:33:53.425519 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.425481 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" event={"ID":"9791f829-4cd3-46c7-8090-a97d6a630584","Type":"ContainerStarted","Data":"c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206"} Jul 24 15:33:53.425695 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.425546 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" event={"ID":"9791f829-4cd3-46c7-8090-a97d6a630584","Type":"ContainerStarted","Data":"f54c0a4226f4a5f77da0516c54fc4ad3d10148fed15465776e151a0548425ade"} Jul 24 15:33:53.452000 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:53.451922 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" podStartSLOduration=1.4518995160000001 podStartE2EDuration="1.451899516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:33:53.44936781 +0000 UTC m=+2252.396861399" watchObservedRunningTime="2026-07-24 15:33:53.451899516 +0000 UTC m=+2252.399393103" Jul 24 15:33:54.076399 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:54.076362 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:54.081211 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:54.081160 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:54.430539 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:54.430508 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:54.431382 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:54.431362 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:33:55.524041 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.524006 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq"] Jul 24 15:33:55.527345 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.527324 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:55.530016 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.529987 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f97fff-serving-cert\"" Jul 24 15:33:55.530133 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.530075 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n8r5x\"" Jul 24 15:33:55.539468 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.539442 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq"] Jul 24 15:33:55.668985 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.668946 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvx6\" (UniqueName: \"kubernetes.io/projected/6d3cdb23-96bc-4897-9370-08c34378b862-kube-api-access-pmvx6\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:55.669164 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.669029 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3cdb23-96bc-4897-9370-08c34378b862-openshift-service-ca\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:55.669164 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.669072 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3cdb23-96bc-4897-9370-08c34378b862-maas-api-tls\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:55.769888 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.769839 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3cdb23-96bc-4897-9370-08c34378b862-openshift-service-ca\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:55.769888 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.769897 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3cdb23-96bc-4897-9370-08c34378b862-maas-api-tls\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:55.770113 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.769955 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvx6\" (UniqueName: \"kubernetes.io/projected/6d3cdb23-96bc-4897-9370-08c34378b862-kube-api-access-pmvx6\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:55.770113 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:33:55.770059 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-serving-cert: secret "maas-api-e2e-api-a-f97fff-serving-cert" not found Jul 24 15:33:55.770219 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:33:55.770133 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3cdb23-96bc-4897-9370-08c34378b862-maas-api-tls podName:6d3cdb23-96bc-4897-9370-08c34378b862 nodeName:}" failed. No retries permitted until 2026-07-24 15:33:56.270116111 +0000 UTC m=+2255.217609674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d3cdb23-96bc-4897-9370-08c34378b862-maas-api-tls") pod "maas-api-e2e-api-a-f97fff-546848b768-2w8jq" (UID: "6d3cdb23-96bc-4897-9370-08c34378b862") : secret "maas-api-e2e-api-a-f97fff-serving-cert" not found Jul 24 15:33:55.770506 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.770489 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3cdb23-96bc-4897-9370-08c34378b862-openshift-service-ca\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:55.781615 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:55.781550 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvx6\" (UniqueName: \"kubernetes.io/projected/6d3cdb23-96bc-4897-9370-08c34378b862-kube-api-access-pmvx6\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:56.275715 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:56.275672 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3cdb23-96bc-4897-9370-08c34378b862-maas-api-tls\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:56.278233 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:56.278209 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3cdb23-96bc-4897-9370-08c34378b862-maas-api-tls\") pod \"maas-api-e2e-api-a-f97fff-546848b768-2w8jq\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:56.438053 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:56.438021 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:56.588441 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:56.588412 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq"] Jul 24 15:33:56.590365 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:33:56.590333 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3cdb23_96bc_4897_9370_08c34378b862.slice/crio-b38d142ea7da5d9b610da350bd444e53dbd55082739c01dc577652c8a10837ce WatchSource:0}: Error finding container b38d142ea7da5d9b610da350bd444e53dbd55082739c01dc577652c8a10837ce: Status 404 returned error can't find the container with id b38d142ea7da5d9b610da350bd444e53dbd55082739c01dc577652c8a10837ce Jul 24 15:33:56.591801 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:56.591780 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:33:57.043395 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.043371 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:33:57.082717 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.082696 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-data\") pod \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " Jul 24 15:33:57.082840 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.082730 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djdg2\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-kube-api-access-djdg2\") pod \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " Jul 24 15:33:57.082940 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.082919 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-certs\") pod \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " Jul 24 15:33:57.083095 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.083070 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-token\") pod \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " Jul 24 15:33:57.083299 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.083154 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75eab9d2-e8b3-4bf0-9382-f6357b34797c" (UID: "75eab9d2-e8b3-4bf0-9382-f6357b34797c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:57.083299 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.083247 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-envoy\") pod \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " Jul 24 15:33:57.083299 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.083283 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-socket\") pod \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " Jul 24 15:33:57.083550 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.083308 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-credential-socket\") pod \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " Jul 24 15:33:57.083550 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.083366 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-podinfo\") pod \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " Jul 24 15:33:57.083550 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.083424 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istiod-ca-cert\") pod \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\" (UID: \"75eab9d2-e8b3-4bf0-9382-f6357b34797c\") " Jul 24 15:33:57.083719 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.083697 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75eab9d2-e8b3-4bf0-9382-f6357b34797c" (UID: "75eab9d2-e8b3-4bf0-9382-f6357b34797c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:57.084040 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.083995 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75eab9d2-e8b3-4bf0-9382-f6357b34797c" (UID: "75eab9d2-e8b3-4bf0-9382-f6357b34797c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:57.084479 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.084457 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-data" (OuterVolumeSpecName: "istio-data") pod "75eab9d2-e8b3-4bf0-9382-f6357b34797c" (UID: "75eab9d2-e8b3-4bf0-9382-f6357b34797c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:57.084618 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.084463 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75eab9d2-e8b3-4bf0-9382-f6357b34797c" (UID: "75eab9d2-e8b3-4bf0-9382-f6357b34797c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:33:57.084726 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.084549 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:57.084909 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.084868 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:57.084909 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.084889 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:57.085665 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.085640 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-kube-api-access-djdg2" (OuterVolumeSpecName: "kube-api-access-djdg2") pod "75eab9d2-e8b3-4bf0-9382-f6357b34797c" (UID: "75eab9d2-e8b3-4bf0-9382-f6357b34797c"). InnerVolumeSpecName "kube-api-access-djdg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:33:57.085864 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.085840 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-token" (OuterVolumeSpecName: "istio-token") pod "75eab9d2-e8b3-4bf0-9382-f6357b34797c" (UID: "75eab9d2-e8b3-4bf0-9382-f6357b34797c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:33:57.086775 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.086746 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75eab9d2-e8b3-4bf0-9382-f6357b34797c" (UID: "75eab9d2-e8b3-4bf0-9382-f6357b34797c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:33:57.086896 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.086875 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75eab9d2-e8b3-4bf0-9382-f6357b34797c" (UID: "75eab9d2-e8b3-4bf0-9382-f6357b34797c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:33:57.186107 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.186075 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:57.186107 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.186103 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:57.186107 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.186113 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:57.186349 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.186122 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:57.186349 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.186131 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djdg2\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-kube-api-access-djdg2\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:57.186349 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.186141 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75eab9d2-e8b3-4bf0-9382-f6357b34797c-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:33:57.445574 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.445545 2557 generic.go:356] "Generic (PLEG): container finished" podID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerID="452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7" exitCode=0 Jul 24 15:33:57.445876 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.445592 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" Jul 24 15:33:57.445876 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.445623 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" event={"ID":"75eab9d2-e8b3-4bf0-9382-f6357b34797c","Type":"ContainerDied","Data":"452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7"} Jul 24 15:33:57.445876 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.445675 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g" event={"ID":"75eab9d2-e8b3-4bf0-9382-f6357b34797c","Type":"ContainerDied","Data":"3b695bc0a8d9590b83ef5ba377ebf470fc774a4cf693ef23270bb87c952f82dc"} Jul 24 15:33:57.445876 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.445696 2557 scope.go:117] "RemoveContainer" containerID="452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7" Jul 24 15:33:57.447403 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.447384 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" event={"ID":"6d3cdb23-96bc-4897-9370-08c34378b862","Type":"ContainerStarted","Data":"4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f"} Jul 24 15:33:57.447516 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.447426 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" event={"ID":"6d3cdb23-96bc-4897-9370-08c34378b862","Type":"ContainerStarted","Data":"b38d142ea7da5d9b610da350bd444e53dbd55082739c01dc577652c8a10837ce"} Jul 24 15:33:57.447588 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.447575 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:33:57.455600 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.455579 2557 scope.go:117] "RemoveContainer" containerID="7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92" Jul 24 15:33:57.463292 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.463272 2557 scope.go:117] "RemoveContainer" containerID="452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7" Jul 24 15:33:57.463558 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:33:57.463540 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7\": container with ID starting with 452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7 not found: ID does not exist" containerID="452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7" Jul 24 15:33:57.463609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.463570 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7"} err="failed to get container status \"452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7\": rpc error: code = NotFound desc = could not find container \"452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7\": container with ID starting with 452e15369a1aa5f0c65f32fc1f4c2a32171c7daa6ae06ebc95208de0e9302aa7 not found: ID does not exist" Jul 24 15:33:57.463609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.463590 2557 scope.go:117] "RemoveContainer" containerID="7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92" Jul 24 15:33:57.463844 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:33:57.463827 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92\": container with ID starting with 7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92 not found: ID does not exist" containerID="7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92" Jul 24 15:33:57.463889 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.463851 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92"} err="failed to get container status \"7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92\": rpc error: code = NotFound desc = could not find container \"7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92\": container with ID starting with 7f31292d06672ca3317782b167d7562d1643801b8f7efa7300cc2b05afd81d92 not found: ID does not exist" Jul 24 15:33:57.473034 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.472991 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" podStartSLOduration=2.00006511 podStartE2EDuration="2.472978799s" totalImagesPullingTime="472.913689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:33:57.470398488 +0000 UTC m=+2256.417892076" watchObservedRunningTime="2026-07-24 15:33:57.472978799 +0000 UTC m=+2256.420472384" Jul 24 15:33:57.486616 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.486591 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g"] Jul 24 15:33:57.491431 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.491410 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dde49b-openshift-default-7d4fcd6cd8-8h54g"] Jul 24 15:33:57.589087 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:33:57.589046 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" path="/var/lib/kubelet/pods/75eab9d2-e8b3-4bf0-9382-f6357b34797c/volumes" Jul 24 15:34:00.730786 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.730737 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89"] Jul 24 15:34:00.731221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.731109 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" Jul 24 15:34:00.731221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.731126 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" Jul 24 15:34:00.731221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.731214 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" Jul 24 15:34:00.735376 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.735352 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.738573 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.738553 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-03bf4f-openshift-default-dockercfg-qv4jf\"" Jul 24 15:34:00.754553 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.754368 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89"] Jul 24 15:34:00.821022 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.820985 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-data\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.821249 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.821042 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44f9j\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-kube-api-access-44f9j\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.821249 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.821105 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-credential-socket\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.821249 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.821142 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-podinfo\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.821249 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.821159 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c9782-b89d-480e-b4eb-35956d373f09-istiod-ca-cert\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.821249 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.821229 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-socket\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.821475 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.821309 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-envoy\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.821475 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.821385 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-certs\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.821475 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.821431 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-token\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922051 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922016 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-data\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922070 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44f9j\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-kube-api-access-44f9j\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922092 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-credential-socket\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922110 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-podinfo\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922128 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c9782-b89d-480e-b4eb-35956d373f09-istiod-ca-cert\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922174 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-socket\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922227 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-envoy\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922254 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-certs\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922358 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922286 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-token\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922740 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922429 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-data\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922740 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922553 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-credential-socket\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922740 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922612 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-certs\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922841 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922795 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-socket\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.922917 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.922900 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c9782-b89d-480e-b4eb-35956d373f09-istiod-ca-cert\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.924823 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.924801 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-envoy\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.925048 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.924921 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-podinfo\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.934446 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.934427 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-token\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:00.934718 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:00.934700 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44f9j\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-kube-api-access-44f9j\") pod \"e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:01.048308 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:01.048225 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:01.216036 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:01.215980 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89"] Jul 24 15:34:01.220015 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:34:01.219984 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9c9782_b89d_480e_b4eb_35956d373f09.slice/crio-6528e752458f6d6ed82f75e3d9d7935c82b73e2d255d2e53b6dd87d51fbecdb9 WatchSource:0}: Error finding container 6528e752458f6d6ed82f75e3d9d7935c82b73e2d255d2e53b6dd87d51fbecdb9: Status 404 returned error can't find the container with id 6528e752458f6d6ed82f75e3d9d7935c82b73e2d255d2e53b6dd87d51fbecdb9 Jul 24 15:34:01.222335 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:01.222304 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:34:01.222413 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:01.222369 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:34:01.222413 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:01.222397 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:34:01.469784 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:01.469747 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" event={"ID":"5b9c9782-b89d-480e-b4eb-35956d373f09","Type":"ContainerStarted","Data":"00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b"} Jul 24 15:34:01.470069 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:01.470044 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" event={"ID":"5b9c9782-b89d-480e-b4eb-35956d373f09","Type":"ContainerStarted","Data":"6528e752458f6d6ed82f75e3d9d7935c82b73e2d255d2e53b6dd87d51fbecdb9"} Jul 24 15:34:01.509897 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:01.509828 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" podStartSLOduration=1.5097908690000001 podStartE2EDuration="1.509790869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:34:01.507848305 +0000 UTC m=+2260.455341892" watchObservedRunningTime="2026-07-24 15:34:01.509790869 +0000 UTC m=+2260.457284456" Jul 24 15:34:02.048681 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:02.048638 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:02.050024 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:02.049997 2557 patch_prober.go:28] interesting pod/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 15:34:02.050091 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:02.050048 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 15:34:03.054336 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.054309 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:03.459442 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.459411 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:34:03.486226 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.485754 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:03.488432 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.488225 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:03.644053 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.644018 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz"] Jul 24 15:34:03.652902 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.645047 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" Jul 24 15:34:03.652902 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.645068 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" Jul 24 15:34:03.652902 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.645714 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="75eab9d2-e8b3-4bf0-9382-f6357b34797c" containerName="istio-proxy" Jul 24 15:34:03.654429 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.654275 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.656839 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.656821 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-03bf4f-serving-cert\"" Jul 24 15:34:03.662215 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.662035 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz"] Jul 24 15:34:03.751014 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.750919 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-maas-api-tls\") pod \"maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.751014 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.750995 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-openshift-service-ca\") pod \"maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.751263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.751053 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59jrf\" (UniqueName: \"kubernetes.io/projected/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-kube-api-access-59jrf\") pod \"maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.851666 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.851634 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-openshift-service-ca\") pod \"maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.851824 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.851695 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59jrf\" (UniqueName: \"kubernetes.io/projected/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-kube-api-access-59jrf\") pod \"maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.851824 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.851789 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-maas-api-tls\") pod \"maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.852478 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.852446 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-openshift-service-ca\") pod \"maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.854776 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.854757 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-maas-api-tls\") pod \"maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.861583 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.861561 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59jrf\" (UniqueName: \"kubernetes.io/projected/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-kube-api-access-59jrf\") pod \"maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:03.968385 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:03.968351 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:04.138414 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:04.138390 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz"] Jul 24 15:34:04.140421 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:34:04.140388 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2ea605_eb99_4aa0_ad39_e278f25a0f25.slice/crio-04a43596847911cc4fcda1b40dea86d3c3af77d9d5a0c026bca9c66bf288404e WatchSource:0}: Error finding container 04a43596847911cc4fcda1b40dea86d3c3af77d9d5a0c026bca9c66bf288404e: Status 404 returned error can't find the container with id 04a43596847911cc4fcda1b40dea86d3c3af77d9d5a0c026bca9c66bf288404e Jul 24 15:34:04.491268 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:04.491232 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" event={"ID":"5e2ea605-eb99-4aa0-ad39-e278f25a0f25","Type":"ContainerStarted","Data":"04a43596847911cc4fcda1b40dea86d3c3af77d9d5a0c026bca9c66bf288404e"} Jul 24 15:34:05.499388 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:05.499352 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" event={"ID":"5e2ea605-eb99-4aa0-ad39-e278f25a0f25","Type":"ContainerStarted","Data":"eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75"} Jul 24 15:34:05.499821 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:05.499419 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:05.519604 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:05.519540 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" podStartSLOduration=2.02580516 podStartE2EDuration="2.519522352s" totalImagesPullingTime="493.717192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:34:05.518727232 +0000 UTC m=+2264.466220818" watchObservedRunningTime="2026-07-24 15:34:05.519522352 +0000 UTC m=+2264.467016001" Jul 24 15:34:07.509525 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:07.509492 2557 generic.go:356] "Generic (PLEG): container finished" podID="9791f829-4cd3-46c7-8090-a97d6a630584" containerID="c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206" exitCode=0 Jul 24 15:34:07.509939 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:07.509551 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" event={"ID":"9791f829-4cd3-46c7-8090-a97d6a630584","Type":"ContainerDied","Data":"c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206"} Jul 24 15:34:07.509939 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:07.509921 2557 scope.go:117] "RemoveContainer" containerID="c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206" Jul 24 15:34:07.511005 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:07.510976 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:34:07.511066 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:07.511047 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:34:07.511101 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:07.511085 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:34:08.519299 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:08.519259 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" event={"ID":"9791f829-4cd3-46c7-8090-a97d6a630584","Type":"ContainerStarted","Data":"a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7"} Jul 24 15:34:09.076112 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:09.075999 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:34:09.077209 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:09.077157 2557 patch_prober.go:28] interesting pod/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 15:34:09.077362 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:09.077249 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 15:34:09.431328 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:09.431283 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:34:10.076795 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:10.076754 2557 patch_prober.go:28] interesting pod/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 15:34:10.077204 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:10.076824 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 15:34:11.076841 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:11.076799 2557 patch_prober.go:28] interesting pod/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 15:34:11.077264 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:11.076873 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 15:34:11.515521 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:11.515482 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:12.076290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:12.076253 2557 patch_prober.go:28] interesting pod/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 15:34:12.076496 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:12.076328 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 15:34:13.080720 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:13.080632 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:34:13.081662 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:13.081642 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:34:16.569178 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:16.569114 2557 generic.go:356] "Generic (PLEG): container finished" podID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerID="00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b" exitCode=0 Jul 24 15:34:16.569729 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:16.569217 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" event={"ID":"5b9c9782-b89d-480e-b4eb-35956d373f09","Type":"ContainerDied","Data":"00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b"} Jul 24 15:34:16.569807 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:16.569744 2557 scope.go:117] "RemoveContainer" containerID="00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b" Jul 24 15:34:16.570804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:16.570778 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:34:16.570894 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:16.570847 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:34:16.570894 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:16.570882 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:34:17.575788 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:17.575753 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" event={"ID":"5b9c9782-b89d-480e-b4eb-35956d373f09","Type":"ContainerStarted","Data":"9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e"} Jul 24 15:34:18.049103 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:18.049074 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:18.050163 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:18.050134 2557 patch_prober.go:28] interesting pod/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 15:34:18.050279 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:18.050215 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 15:34:18.485937 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:18.485904 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:19.049254 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:19.049223 2557 patch_prober.go:28] interesting pod/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 15:34:19.049633 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:19.049276 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 15:34:20.049581 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:20.049541 2557 patch_prober.go:28] interesting pod/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 15:34:20.050054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:20.049607 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 15:34:21.048830 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:21.048796 2557 patch_prober.go:28] interesting pod/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 15:34:21.049008 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:21.048849 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 15:34:21.729553 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:21.729521 2557 scope.go:117] "RemoveContainer" containerID="44f80be966570be4ca0809b2f1d80be247be92938fc2b39bdad6533899745279" Jul 24 15:34:22.052703 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:22.052622 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:22.053469 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:22.053452 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:34:31.863439 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:31.863407 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq"] Jul 24 15:34:31.863897 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:31.863636 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" podUID="6d3cdb23-96bc-4897-9370-08c34378b862" containerName="maas-api" containerID="cri-o://4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f" gracePeriod=30 Jul 24 15:34:32.125753 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.125729 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:34:32.243119 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.243084 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3cdb23-96bc-4897-9370-08c34378b862-maas-api-tls\") pod \"6d3cdb23-96bc-4897-9370-08c34378b862\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " Jul 24 15:34:32.243331 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.243130 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3cdb23-96bc-4897-9370-08c34378b862-openshift-service-ca\") pod \"6d3cdb23-96bc-4897-9370-08c34378b862\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " Jul 24 15:34:32.243331 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.243171 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmvx6\" (UniqueName: \"kubernetes.io/projected/6d3cdb23-96bc-4897-9370-08c34378b862-kube-api-access-pmvx6\") pod \"6d3cdb23-96bc-4897-9370-08c34378b862\" (UID: \"6d3cdb23-96bc-4897-9370-08c34378b862\") " Jul 24 15:34:32.243541 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.243512 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3cdb23-96bc-4897-9370-08c34378b862-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d3cdb23-96bc-4897-9370-08c34378b862" (UID: "6d3cdb23-96bc-4897-9370-08c34378b862"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:34:32.245324 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.245298 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3cdb23-96bc-4897-9370-08c34378b862-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d3cdb23-96bc-4897-9370-08c34378b862" (UID: "6d3cdb23-96bc-4897-9370-08c34378b862"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:34:32.245464 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.245348 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3cdb23-96bc-4897-9370-08c34378b862-kube-api-access-pmvx6" (OuterVolumeSpecName: "kube-api-access-pmvx6") pod "6d3cdb23-96bc-4897-9370-08c34378b862" (UID: "6d3cdb23-96bc-4897-9370-08c34378b862"). InnerVolumeSpecName "kube-api-access-pmvx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:34:32.344384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.344350 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3cdb23-96bc-4897-9370-08c34378b862-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:32.344384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.344380 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmvx6\" (UniqueName: \"kubernetes.io/projected/6d3cdb23-96bc-4897-9370-08c34378b862-kube-api-access-pmvx6\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:32.344384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.344390 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d3cdb23-96bc-4897-9370-08c34378b862-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:32.645274 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.645244 2557 generic.go:356] "Generic (PLEG): container finished" podID="6d3cdb23-96bc-4897-9370-08c34378b862" containerID="4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f" exitCode=0 Jul 24 15:34:32.645444 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.645288 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" Jul 24 15:34:32.645444 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.645323 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" event={"ID":"6d3cdb23-96bc-4897-9370-08c34378b862","Type":"ContainerDied","Data":"4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f"} Jul 24 15:34:32.645444 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.645376 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq" event={"ID":"6d3cdb23-96bc-4897-9370-08c34378b862","Type":"ContainerDied","Data":"b38d142ea7da5d9b610da350bd444e53dbd55082739c01dc577652c8a10837ce"} Jul 24 15:34:32.645444 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.645398 2557 scope.go:117] "RemoveContainer" containerID="4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f" Jul 24 15:34:32.654935 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.654920 2557 scope.go:117] "RemoveContainer" containerID="4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f" Jul 24 15:34:32.655215 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:34:32.655172 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f\": container with ID starting with 4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f not found: ID does not exist" containerID="4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f" Jul 24 15:34:32.655274 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.655228 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f"} err="failed to get container status \"4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f\": rpc error: code = NotFound desc = could not find container \"4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f\": container with ID starting with 4efd2d5533d90d55c56136c45f63bd2060b83cb5f82409e1d8568e11cab2558f not found: ID does not exist" Jul 24 15:34:32.668798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.668766 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq"] Jul 24 15:34:32.672855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:32.672824 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f97fff-546848b768-2w8jq"] Jul 24 15:34:33.587283 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:33.587253 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3cdb23-96bc-4897-9370-08c34378b862" path="/var/lib/kubelet/pods/6d3cdb23-96bc-4897-9370-08c34378b862/volumes" Jul 24 15:34:43.165694 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:43.165647 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw"] Jul 24 15:34:43.166179 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:43.165917 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" containerID="cri-o://a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7" gracePeriod=30 Jul 24 15:34:48.419353 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.419328 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:34:48.482750 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.482672 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-istio-token\") pod \"9791f829-4cd3-46c7-8090-a97d6a630584\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " Jul 24 15:34:48.482750 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.482708 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-credential-socket\") pod \"9791f829-4cd3-46c7-8090-a97d6a630584\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " Jul 24 15:34:48.482750 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.482729 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9791f829-4cd3-46c7-8090-a97d6a630584-istiod-ca-cert\") pod \"9791f829-4cd3-46c7-8090-a97d6a630584\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " Jul 24 15:34:48.482750 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.482748 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tctm8\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-kube-api-access-tctm8\") pod \"9791f829-4cd3-46c7-8090-a97d6a630584\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " Jul 24 15:34:48.483078 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.482810 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-envoy\") pod \"9791f829-4cd3-46c7-8090-a97d6a630584\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " Jul 24 15:34:48.483078 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.482831 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-data\") pod \"9791f829-4cd3-46c7-8090-a97d6a630584\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " Jul 24 15:34:48.483078 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.482847 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-certs\") pod \"9791f829-4cd3-46c7-8090-a97d6a630584\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " Jul 24 15:34:48.483078 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483018 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9791f829-4cd3-46c7-8090-a97d6a630584" (UID: "9791f829-4cd3-46c7-8090-a97d6a630584"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:34:48.483316 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483111 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-socket\") pod \"9791f829-4cd3-46c7-8090-a97d6a630584\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " Jul 24 15:34:48.483316 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483152 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9791f829-4cd3-46c7-8090-a97d6a630584" (UID: "9791f829-4cd3-46c7-8090-a97d6a630584"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:34:48.483316 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483166 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9791f829-4cd3-46c7-8090-a97d6a630584-istio-podinfo\") pod \"9791f829-4cd3-46c7-8090-a97d6a630584\" (UID: \"9791f829-4cd3-46c7-8090-a97d6a630584\") " Jul 24 15:34:48.483316 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483171 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9791f829-4cd3-46c7-8090-a97d6a630584-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9791f829-4cd3-46c7-8090-a97d6a630584" (UID: "9791f829-4cd3-46c7-8090-a97d6a630584"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:34:48.483504 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483384 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9791f829-4cd3-46c7-8090-a97d6a630584" (UID: "9791f829-4cd3-46c7-8090-a97d6a630584"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:34:48.483563 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483518 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:48.483563 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483538 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:48.483563 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483553 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:48.483676 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.483567 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9791f829-4cd3-46c7-8090-a97d6a630584-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:48.484089 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.484061 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-data" (OuterVolumeSpecName: "istio-data") pod "9791f829-4cd3-46c7-8090-a97d6a630584" (UID: "9791f829-4cd3-46c7-8090-a97d6a630584"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:34:48.485097 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.485075 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-istio-token" (OuterVolumeSpecName: "istio-token") pod "9791f829-4cd3-46c7-8090-a97d6a630584" (UID: "9791f829-4cd3-46c7-8090-a97d6a630584"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:34:48.485284 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.485269 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9791f829-4cd3-46c7-8090-a97d6a630584-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9791f829-4cd3-46c7-8090-a97d6a630584" (UID: "9791f829-4cd3-46c7-8090-a97d6a630584"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:34:48.485597 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.485577 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9791f829-4cd3-46c7-8090-a97d6a630584" (UID: "9791f829-4cd3-46c7-8090-a97d6a630584"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:34:48.485663 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.485580 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-kube-api-access-tctm8" (OuterVolumeSpecName: "kube-api-access-tctm8") pod "9791f829-4cd3-46c7-8090-a97d6a630584" (UID: "9791f829-4cd3-46c7-8090-a97d6a630584"). InnerVolumeSpecName "kube-api-access-tctm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:34:48.584718 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.584683 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9791f829-4cd3-46c7-8090-a97d6a630584-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:48.584718 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.584713 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:48.584718 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.584722 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tctm8\" (UniqueName: \"kubernetes.io/projected/9791f829-4cd3-46c7-8090-a97d6a630584-kube-api-access-tctm8\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:48.584953 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.584731 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:48.584953 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.584739 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9791f829-4cd3-46c7-8090-a97d6a630584-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:48.718965 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.718931 2557 generic.go:356] "Generic (PLEG): container finished" podID="9791f829-4cd3-46c7-8090-a97d6a630584" containerID="a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7" exitCode=0 Jul 24 15:34:48.719133 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.718979 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" Jul 24 15:34:48.719133 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.718998 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" event={"ID":"9791f829-4cd3-46c7-8090-a97d6a630584","Type":"ContainerDied","Data":"a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7"} Jul 24 15:34:48.719133 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.719042 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw" event={"ID":"9791f829-4cd3-46c7-8090-a97d6a630584","Type":"ContainerDied","Data":"f54c0a4226f4a5f77da0516c54fc4ad3d10148fed15465776e151a0548425ade"} Jul 24 15:34:48.719133 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.719063 2557 scope.go:117] "RemoveContainer" containerID="a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7" Jul 24 15:34:48.728028 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.728009 2557 scope.go:117] "RemoveContainer" containerID="c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206" Jul 24 15:34:48.735809 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.735790 2557 scope.go:117] "RemoveContainer" containerID="a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7" Jul 24 15:34:48.736047 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:34:48.736028 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7\": container with ID starting with a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7 not found: ID does not exist" containerID="a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7" Jul 24 15:34:48.736087 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.736056 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7"} err="failed to get container status \"a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7\": rpc error: code = NotFound desc = could not find container \"a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7\": container with ID starting with a7deb79a5bac441cd36398b4a3b88f4f45fa64baec63dd37ae0badde35866cd7 not found: ID does not exist" Jul 24 15:34:48.736087 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.736073 2557 scope.go:117] "RemoveContainer" containerID="c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206" Jul 24 15:34:48.736325 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:34:48.736306 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206\": container with ID starting with c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206 not found: ID does not exist" containerID="c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206" Jul 24 15:34:48.736373 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.736331 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206"} err="failed to get container status \"c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206\": rpc error: code = NotFound desc = could not find container \"c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206\": container with ID starting with c5abeb33f077a4abf184c4e135c4b7e6206b6dd025e4fe6a91c6bb3c63772206 not found: ID does not exist" Jul 24 15:34:48.741382 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.741359 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw"] Jul 24 15:34:48.750584 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:48.750559 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f97fff-openshift-default-5fc554fbf6-p6pzw"] Jul 24 15:34:49.588340 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:49.588309 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" path="/var/lib/kubelet/pods/9791f829-4cd3-46c7-8090-a97d6a630584/volumes" Jul 24 15:34:58.980926 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:58.980893 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz"] Jul 24 15:34:58.981346 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:58.981118 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" podUID="5e2ea605-eb99-4aa0-ad39-e278f25a0f25" containerName="maas-api" containerID="cri-o://eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75" gracePeriod=30 Jul 24 15:34:59.230386 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.230357 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:59.276489 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.276409 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59jrf\" (UniqueName: \"kubernetes.io/projected/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-kube-api-access-59jrf\") pod \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " Jul 24 15:34:59.276489 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.276472 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-maas-api-tls\") pod \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " Jul 24 15:34:59.276677 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.276520 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-openshift-service-ca\") pod \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\" (UID: \"5e2ea605-eb99-4aa0-ad39-e278f25a0f25\") " Jul 24 15:34:59.276856 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.276834 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e2ea605-eb99-4aa0-ad39-e278f25a0f25" (UID: "5e2ea605-eb99-4aa0-ad39-e278f25a0f25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:34:59.278610 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.278584 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e2ea605-eb99-4aa0-ad39-e278f25a0f25" (UID: "5e2ea605-eb99-4aa0-ad39-e278f25a0f25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:34:59.278726 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.278651 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-kube-api-access-59jrf" (OuterVolumeSpecName: "kube-api-access-59jrf") pod "5e2ea605-eb99-4aa0-ad39-e278f25a0f25" (UID: "5e2ea605-eb99-4aa0-ad39-e278f25a0f25"). InnerVolumeSpecName "kube-api-access-59jrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:34:59.377359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.377330 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59jrf\" (UniqueName: \"kubernetes.io/projected/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-kube-api-access-59jrf\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:59.377359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.377356 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:59.377359 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.377365 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2ea605-eb99-4aa0-ad39-e278f25a0f25-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:34:59.784552 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.784503 2557 generic.go:356] "Generic (PLEG): container finished" podID="5e2ea605-eb99-4aa0-ad39-e278f25a0f25" containerID="eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75" exitCode=0 Jul 24 15:34:59.784552 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.784550 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" Jul 24 15:34:59.784552 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.784559 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" event={"ID":"5e2ea605-eb99-4aa0-ad39-e278f25a0f25","Type":"ContainerDied","Data":"eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75"} Jul 24 15:34:59.785086 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.784614 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz" event={"ID":"5e2ea605-eb99-4aa0-ad39-e278f25a0f25","Type":"ContainerDied","Data":"04a43596847911cc4fcda1b40dea86d3c3af77d9d5a0c026bca9c66bf288404e"} Jul 24 15:34:59.785086 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.784646 2557 scope.go:117] "RemoveContainer" containerID="eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75" Jul 24 15:34:59.793243 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.793226 2557 scope.go:117] "RemoveContainer" containerID="eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75" Jul 24 15:34:59.793499 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:34:59.793481 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75\": container with ID starting with eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75 not found: ID does not exist" containerID="eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75" Jul 24 15:34:59.793567 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.793510 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75"} err="failed to get container status \"eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75\": rpc error: code = NotFound desc = could not find container \"eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75\": container with ID starting with eba07cf5659cc4d93af9a91dd1e3748420a2cb12b10416a99c87c6a14ce51d75 not found: ID does not exist" Jul 24 15:34:59.802491 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.802471 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz"] Jul 24 15:34:59.806794 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:34:59.806776 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-03bf4f-5584d44f76-ldgrz"] Jul 24 15:35:01.587916 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:01.587887 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e2ea605-eb99-4aa0-ad39-e278f25a0f25" path="/var/lib/kubelet/pods/5e2ea605-eb99-4aa0-ad39-e278f25a0f25/volumes" Jul 24 15:35:10.095096 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:10.095064 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89"] Jul 24 15:35:10.095716 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:10.095342 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" containerID="cri-o://9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e" gracePeriod=30 Jul 24 15:35:11.003582 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.003548 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb"] Jul 24 15:35:11.003889 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.003876 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" Jul 24 15:35:11.003939 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.003890 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" Jul 24 15:35:11.003939 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.003902 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" Jul 24 15:35:11.003939 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.003908 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" Jul 24 15:35:11.003939 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.003919 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2ea605-eb99-4aa0-ad39-e278f25a0f25" containerName="maas-api" Jul 24 15:35:11.003939 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.003925 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2ea605-eb99-4aa0-ad39-e278f25a0f25" containerName="maas-api" Jul 24 15:35:11.003939 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.003935 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3cdb23-96bc-4897-9370-08c34378b862" containerName="maas-api" Jul 24 15:35:11.003939 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.003940 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3cdb23-96bc-4897-9370-08c34378b862" containerName="maas-api" Jul 24 15:35:11.004149 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.004005 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" Jul 24 15:35:11.004149 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.004012 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9791f829-4cd3-46c7-8090-a97d6a630584" containerName="istio-proxy" Jul 24 15:35:11.004149 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.004018 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3cdb23-96bc-4897-9370-08c34378b862" containerName="maas-api" Jul 24 15:35:11.004149 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.004025 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2ea605-eb99-4aa0-ad39-e278f25a0f25" containerName="maas-api" Jul 24 15:35:11.005901 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.005886 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.008777 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.008755 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-d419ce-openshift-default-dockercfg-7g5ls\"" Jul 24 15:35:11.022288 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.022269 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb"] Jul 24 15:35:11.180862 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.180825 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-credential-socket\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.181319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.180873 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-socket\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.181319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.180892 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-data\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.181319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.180961 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-certs\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.181319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.181012 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-envoy\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.181319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.181053 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-podinfo\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.181319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.181104 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtqr\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-kube-api-access-ndtqr\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.181319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.181139 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-token\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.181319 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.181163 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istiod-ca-cert\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282355 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282261 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istiod-ca-cert\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282355 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282305 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-credential-socket\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282355 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282333 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-socket\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282355 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282356 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-data\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282677 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282390 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-certs\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282677 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282426 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-envoy\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282677 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-podinfo\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282677 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282519 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtqr\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-kube-api-access-ndtqr\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282677 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282584 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-token\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282995 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282753 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-socket\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282995 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282813 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-credential-socket\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.282995 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.282867 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-certs\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.283161 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.283055 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-data\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.283161 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.283118 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istiod-ca-cert\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.284820 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.284797 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-envoy\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.285285 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.285265 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-podinfo\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.294262 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.294234 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtqr\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-kube-api-access-ndtqr\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.294366 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.294265 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-token\") pod \"e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.316443 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.316407 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:11.448385 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.448359 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb"] Jul 24 15:35:11.449896 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:35:11.449873 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58ec4dc_a49b_4c33_a9e9_a846d418d9e4.slice/crio-1b349da1da8820e7bcaa904b5524e42c82531c987bfe6694f938f90af807d8fb WatchSource:0}: Error finding container 1b349da1da8820e7bcaa904b5524e42c82531c987bfe6694f938f90af807d8fb: Status 404 returned error can't find the container with id 1b349da1da8820e7bcaa904b5524e42c82531c987bfe6694f938f90af807d8fb Jul 24 15:35:11.452023 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.451995 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:11.452080 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.452062 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:11.452119 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.452090 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:11.839080 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.839039 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" event={"ID":"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4","Type":"ContainerStarted","Data":"f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1"} Jul 24 15:35:11.839280 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.839105 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" event={"ID":"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4","Type":"ContainerStarted","Data":"1b349da1da8820e7bcaa904b5524e42c82531c987bfe6694f938f90af807d8fb"} Jul 24 15:35:11.862170 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:11.862128 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" podStartSLOduration=1.862108644 podStartE2EDuration="1.862108644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:35:11.860761793 +0000 UTC m=+2330.808255382" watchObservedRunningTime="2026-07-24 15:35:11.862108644 +0000 UTC m=+2330.809602231" Jul 24 15:35:12.317012 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:12.316976 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:12.321975 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:12.321940 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:12.844247 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:12.844216 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:12.845301 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:12.845281 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:13.895836 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:13.895806 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj"] Jul 24 15:35:13.898137 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:13.898117 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:13.900632 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:13.900615 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n8r5x\"" Jul 24 15:35:13.900725 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:13.900615 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d419ce-serving-cert\"" Jul 24 15:35:13.908617 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:13.908594 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj"] Jul 24 15:35:14.008591 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.008545 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3ff614-b21c-4800-8d5d-3eb3e69df761-maas-api-tls\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.008591 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.008595 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ff614-b21c-4800-8d5d-3eb3e69df761-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.008868 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.008697 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdc46\" (UniqueName: \"kubernetes.io/projected/8c3ff614-b21c-4800-8d5d-3eb3e69df761-kube-api-access-rdc46\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.109875 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.109843 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdc46\" (UniqueName: \"kubernetes.io/projected/8c3ff614-b21c-4800-8d5d-3eb3e69df761-kube-api-access-rdc46\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.110032 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.109923 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3ff614-b21c-4800-8d5d-3eb3e69df761-maas-api-tls\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.110032 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.109942 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ff614-b21c-4800-8d5d-3eb3e69df761-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.110111 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:35:14.110063 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-serving-cert: secret "maas-api-e2e-shared-a-d419ce-serving-cert" not found Jul 24 15:35:14.110162 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:35:14.110146 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c3ff614-b21c-4800-8d5d-3eb3e69df761-maas-api-tls podName:8c3ff614-b21c-4800-8d5d-3eb3e69df761 nodeName:}" failed. No retries permitted until 2026-07-24 15:35:14.610125 +0000 UTC m=+2333.557618567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c3ff614-b21c-4800-8d5d-3eb3e69df761-maas-api-tls") pod "maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" (UID: "8c3ff614-b21c-4800-8d5d-3eb3e69df761") : secret "maas-api-e2e-shared-a-d419ce-serving-cert" not found Jul 24 15:35:14.110747 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.110726 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ff614-b21c-4800-8d5d-3eb3e69df761-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.119797 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.119771 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdc46\" (UniqueName: \"kubernetes.io/projected/8c3ff614-b21c-4800-8d5d-3eb3e69df761-kube-api-access-rdc46\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.614002 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.613966 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3ff614-b21c-4800-8d5d-3eb3e69df761-maas-api-tls\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.616500 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.616479 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3ff614-b21c-4800-8d5d-3eb3e69df761-maas-api-tls\") pod \"maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.809317 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.809283 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:14.949361 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:14.949336 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj"] Jul 24 15:35:14.951510 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:35:14.951467 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3ff614_b21c_4800_8d5d_3eb3e69df761.slice/crio-946315060e4d2f353bc4ea56b3377aa3c98d2e85845a8ee86882ee451a8f7f46 WatchSource:0}: Error finding container 946315060e4d2f353bc4ea56b3377aa3c98d2e85845a8ee86882ee451a8f7f46: Status 404 returned error can't find the container with id 946315060e4d2f353bc4ea56b3377aa3c98d2e85845a8ee86882ee451a8f7f46 Jul 24 15:35:15.342012 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.341988 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:35:15.524344 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524225 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-certs\") pod \"5b9c9782-b89d-480e-b4eb-35956d373f09\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " Jul 24 15:35:15.524489 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524413 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-podinfo\") pod \"5b9c9782-b89d-480e-b4eb-35956d373f09\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " Jul 24 15:35:15.524489 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524460 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c9782-b89d-480e-b4eb-35956d373f09-istiod-ca-cert\") pod \"5b9c9782-b89d-480e-b4eb-35956d373f09\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " Jul 24 15:35:15.524581 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524484 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b9c9782-b89d-480e-b4eb-35956d373f09" (UID: "5b9c9782-b89d-480e-b4eb-35956d373f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:35:15.524581 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524488 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-envoy\") pod \"5b9c9782-b89d-480e-b4eb-35956d373f09\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " Jul 24 15:35:15.524581 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524541 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44f9j\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-kube-api-access-44f9j\") pod \"5b9c9782-b89d-480e-b4eb-35956d373f09\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " Jul 24 15:35:15.524701 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524591 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-data\") pod \"5b9c9782-b89d-480e-b4eb-35956d373f09\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " Jul 24 15:35:15.524701 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524610 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-token\") pod \"5b9c9782-b89d-480e-b4eb-35956d373f09\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " Jul 24 15:35:15.524701 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524653 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-socket\") pod \"5b9c9782-b89d-480e-b4eb-35956d373f09\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " Jul 24 15:35:15.524701 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524689 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-credential-socket\") pod \"5b9c9782-b89d-480e-b4eb-35956d373f09\" (UID: \"5b9c9782-b89d-480e-b4eb-35956d373f09\") " Jul 24 15:35:15.524915 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.524875 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9c9782-b89d-480e-b4eb-35956d373f09-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b9c9782-b89d-480e-b4eb-35956d373f09" (UID: "5b9c9782-b89d-480e-b4eb-35956d373f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:35:15.525138 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.525061 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b9c9782-b89d-480e-b4eb-35956d373f09" (UID: "5b9c9782-b89d-480e-b4eb-35956d373f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:35:15.525367 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.525128 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b9c9782-b89d-480e-b4eb-35956d373f09" (UID: "5b9c9782-b89d-480e-b4eb-35956d373f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:35:15.525367 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.525334 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:35:15.525367 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.525356 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c9782-b89d-480e-b4eb-35956d373f09-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:35:15.525786 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.525754 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b9c9782-b89d-480e-b4eb-35956d373f09" (UID: "5b9c9782-b89d-480e-b4eb-35956d373f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:35:15.527001 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.526977 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b9c9782-b89d-480e-b4eb-35956d373f09" (UID: "5b9c9782-b89d-480e-b4eb-35956d373f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:35:15.527105 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.527084 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b9c9782-b89d-480e-b4eb-35956d373f09" (UID: "5b9c9782-b89d-480e-b4eb-35956d373f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:35:15.527172 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.527157 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-kube-api-access-44f9j" (OuterVolumeSpecName: "kube-api-access-44f9j") pod "5b9c9782-b89d-480e-b4eb-35956d373f09" (UID: "5b9c9782-b89d-480e-b4eb-35956d373f09"). InnerVolumeSpecName "kube-api-access-44f9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:35:15.527285 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.527166 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b9c9782-b89d-480e-b4eb-35956d373f09" (UID: "5b9c9782-b89d-480e-b4eb-35956d373f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:35:15.626242 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.626215 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:35:15.626242 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.626238 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:35:15.626410 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.626248 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44f9j\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-kube-api-access-44f9j\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:35:15.626410 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.626257 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:35:15.626410 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.626265 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9c9782-b89d-480e-b4eb-35956d373f09-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:35:15.626410 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.626274 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:35:15.626410 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.626282 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9c9782-b89d-480e-b4eb-35956d373f09-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:35:15.859725 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.859617 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" event={"ID":"8c3ff614-b21c-4800-8d5d-3eb3e69df761","Type":"ContainerStarted","Data":"63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b"} Jul 24 15:35:15.859725 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.859670 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" event={"ID":"8c3ff614-b21c-4800-8d5d-3eb3e69df761","Type":"ContainerStarted","Data":"946315060e4d2f353bc4ea56b3377aa3c98d2e85845a8ee86882ee451a8f7f46"} Jul 24 15:35:15.859725 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.859698 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:15.861292 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.861268 2557 generic.go:356] "Generic (PLEG): container finished" podID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerID="9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e" exitCode=0 Jul 24 15:35:15.861432 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.861306 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" event={"ID":"5b9c9782-b89d-480e-b4eb-35956d373f09","Type":"ContainerDied","Data":"9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e"} Jul 24 15:35:15.861432 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.861308 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" Jul 24 15:35:15.861432 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.861339 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89" event={"ID":"5b9c9782-b89d-480e-b4eb-35956d373f09","Type":"ContainerDied","Data":"6528e752458f6d6ed82f75e3d9d7935c82b73e2d255d2e53b6dd87d51fbecdb9"} Jul 24 15:35:15.861432 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.861361 2557 scope.go:117] "RemoveContainer" containerID="9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e" Jul 24 15:35:15.869748 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.869729 2557 scope.go:117] "RemoveContainer" containerID="00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b" Jul 24 15:35:15.877610 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.877589 2557 scope.go:117] "RemoveContainer" containerID="9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e" Jul 24 15:35:15.877867 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:35:15.877847 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e\": container with ID starting with 9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e not found: ID does not exist" containerID="9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e" Jul 24 15:35:15.877934 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.877876 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e"} err="failed to get container status \"9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e\": rpc error: code = NotFound desc = could not find container \"9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e\": container with ID starting with 9863254fa2add579acb0034cbfd74b6a20bf8b7aa2ee9f3d5a3bfabdeebfbf9e not found: ID does not exist" Jul 24 15:35:15.877934 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.877891 2557 scope.go:117] "RemoveContainer" containerID="00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b" Jul 24 15:35:15.878160 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:35:15.878113 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b\": container with ID starting with 00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b not found: ID does not exist" containerID="00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b" Jul 24 15:35:15.878246 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.878170 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b"} err="failed to get container status \"00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b\": rpc error: code = NotFound desc = could not find container \"00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b\": container with ID starting with 00453a473cc143852986a1a75fed405c8e6d14f35aabc097abc98a6bc768447b not found: ID does not exist" Jul 24 15:35:15.878851 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.878812 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" podStartSLOduration=2.480547061 podStartE2EDuration="2.878798741s" totalImagesPullingTime="398.25168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:35:15.877037301 +0000 UTC m=+2334.824530890" watchObservedRunningTime="2026-07-24 15:35:15.878798741 +0000 UTC m=+2334.826292325" Jul 24 15:35:15.890397 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.890376 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89"] Jul 24 15:35:15.894488 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:15.894470 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-03bf4f-openshift-default-d96fbffd4-j2v89"] Jul 24 15:35:17.587260 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:17.587231 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" path="/var/lib/kubelet/pods/5b9c9782-b89d-480e-b4eb-35956d373f09/volumes" Jul 24 15:35:18.972586 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:18.972547 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m"] Jul 24 15:35:18.972942 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:18.972929 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" Jul 24 15:35:18.972942 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:18.972942 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" Jul 24 15:35:18.973021 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:18.973003 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" Jul 24 15:35:18.975889 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:18.975869 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:18.978763 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:18.978744 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6b942c-openshift-default-dockercfg-s8rnc\"" Jul 24 15:35:18.988770 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:18.988744 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m"] Jul 24 15:35:19.055587 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.055552 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-socket\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.055587 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.055588 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-credential-socket\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.055798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.055606 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-podinfo\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.055798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.055632 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-data\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.055798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.055649 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-envoy\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.055798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.055729 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-token\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.055798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.055777 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgms6\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-kube-api-access-bgms6\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.055798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.055797 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-certs\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.055980 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.055828 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istiod-ca-cert\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.156875 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.156819 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-socket\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.156875 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.156877 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-credential-socket\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157112 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157001 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-podinfo\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157112 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157056 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-data\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157112 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157081 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-envoy\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157287 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157125 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-token\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157287 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157170 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgms6\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-kube-api-access-bgms6\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157287 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-certs\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157287 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157248 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istiod-ca-cert\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157287 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157275 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-credential-socket\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157495 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157452 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-socket\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157668 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157636 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-data\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157821 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157682 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-certs\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.157933 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.157917 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istiod-ca-cert\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.159521 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.159502 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-envoy\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.159748 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.159732 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-podinfo\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.166770 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.166743 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-token\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.166916 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.166900 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgms6\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-kube-api-access-bgms6\") pod \"e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.287446 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.287367 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:19.430618 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.430558 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m"] Jul 24 15:35:19.432695 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:35:19.432668 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc9d46d_cd70_42d0_87d9_97b9e750375e.slice/crio-5501a50f285bc2185f321f54b0241b83a9acd57825d1636139fae5eb51daaadf WatchSource:0}: Error finding container 5501a50f285bc2185f321f54b0241b83a9acd57825d1636139fae5eb51daaadf: Status 404 returned error can't find the container with id 5501a50f285bc2185f321f54b0241b83a9acd57825d1636139fae5eb51daaadf Jul 24 15:35:19.435073 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.435043 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:19.435145 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.435107 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:19.435145 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.435136 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:19.881081 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.881046 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" event={"ID":"ecc9d46d-cd70-42d0-87d9-97b9e750375e","Type":"ContainerStarted","Data":"6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec"} Jul 24 15:35:19.881289 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:19.881096 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" event={"ID":"ecc9d46d-cd70-42d0-87d9-97b9e750375e","Type":"ContainerStarted","Data":"5501a50f285bc2185f321f54b0241b83a9acd57825d1636139fae5eb51daaadf"} Jul 24 15:35:20.287558 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:20.287525 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:20.292662 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:20.292636 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:20.315027 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:20.314972 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" podStartSLOduration=2.31495463 podStartE2EDuration="2.31495463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:35:19.902420649 +0000 UTC m=+2338.849914235" watchObservedRunningTime="2026-07-24 15:35:20.31495463 +0000 UTC m=+2339.262448216" Jul 24 15:35:20.885833 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:20.885803 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:20.886845 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:20.886828 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:21.872619 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.872591 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:35:21.962478 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.962439 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk"] Jul 24 15:35:21.962834 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.962819 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" Jul 24 15:35:21.962834 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.962837 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" Jul 24 15:35:21.962949 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.962901 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9c9782-b89d-480e-b4eb-35956d373f09" containerName="istio-proxy" Jul 24 15:35:21.966021 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.965998 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:21.968834 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.968808 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-6b942c-serving-cert\"" Jul 24 15:35:21.978792 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.978759 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk"] Jul 24 15:35:21.983709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.983688 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951c948c-9aba-4b39-9584-6d6b713a7859-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:21.983813 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.983717 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n849\" (UniqueName: \"kubernetes.io/projected/951c948c-9aba-4b39-9584-6d6b713a7859-kube-api-access-4n849\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:21.983951 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:21.983928 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951c948c-9aba-4b39-9584-6d6b713a7859-maas-api-tls\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:22.089258 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:22.086639 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951c948c-9aba-4b39-9584-6d6b713a7859-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:22.089258 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:22.086693 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n849\" (UniqueName: \"kubernetes.io/projected/951c948c-9aba-4b39-9584-6d6b713a7859-kube-api-access-4n849\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:22.089258 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:22.086758 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951c948c-9aba-4b39-9584-6d6b713a7859-maas-api-tls\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:22.089258 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:35:22.086942 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-serving-cert: secret "maas-api-e2e-shared-b-6b942c-serving-cert" not found Jul 24 15:35:22.089258 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:35:22.087003 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/951c948c-9aba-4b39-9584-6d6b713a7859-maas-api-tls podName:951c948c-9aba-4b39-9584-6d6b713a7859 nodeName:}" failed. No retries permitted until 2026-07-24 15:35:22.586984362 +0000 UTC m=+2341.534477940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/951c948c-9aba-4b39-9584-6d6b713a7859-maas-api-tls") pod "maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" (UID: "951c948c-9aba-4b39-9584-6d6b713a7859") : secret "maas-api-e2e-shared-b-6b942c-serving-cert" not found Jul 24 15:35:22.089258 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:22.087979 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951c948c-9aba-4b39-9584-6d6b713a7859-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:22.097168 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:22.097140 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n849\" (UniqueName: \"kubernetes.io/projected/951c948c-9aba-4b39-9584-6d6b713a7859-kube-api-access-4n849\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:22.591302 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:22.591271 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951c948c-9aba-4b39-9584-6d6b713a7859-maas-api-tls\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:22.593863 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:22.593839 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951c948c-9aba-4b39-9584-6d6b713a7859-maas-api-tls\") pod \"maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:22.876633 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:22.876543 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:23.076657 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:23.076631 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk"] Jul 24 15:35:23.079114 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:35:23.079085 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951c948c_9aba_4b39_9584_6d6b713a7859.slice/crio-79a4051ac8ec44c4dbba1c86819f3f0a5c15146ae5dbe126e0806ff05c8435f3 WatchSource:0}: Error finding container 79a4051ac8ec44c4dbba1c86819f3f0a5c15146ae5dbe126e0806ff05c8435f3: Status 404 returned error can't find the container with id 79a4051ac8ec44c4dbba1c86819f3f0a5c15146ae5dbe126e0806ff05c8435f3 Jul 24 15:35:23.906750 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:23.906699 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" event={"ID":"951c948c-9aba-4b39-9584-6d6b713a7859","Type":"ContainerStarted","Data":"79a4051ac8ec44c4dbba1c86819f3f0a5c15146ae5dbe126e0806ff05c8435f3"} Jul 24 15:35:24.917667 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:24.917626 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" event={"ID":"951c948c-9aba-4b39-9584-6d6b713a7859","Type":"ContainerStarted","Data":"7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781"} Jul 24 15:35:24.918282 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:24.918260 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:24.937061 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:24.937013 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" podStartSLOduration=2.4510134470000002 podStartE2EDuration="3.93699743s" totalImagesPullingTime="1.485983983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:35:24.935305099 +0000 UTC m=+2343.882798686" watchObservedRunningTime="2026-07-24 15:35:24.93699743 +0000 UTC m=+2343.884491016" Jul 24 15:35:28.886367 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:28.886319 2557 patch_prober.go:28] interesting pod/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 15:35:28.886889 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:28.886446 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 15:35:30.928073 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:30.928035 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:35:30.946050 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:30.946019 2557 generic.go:356] "Generic (PLEG): container finished" podID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerID="f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1" exitCode=0 Jul 24 15:35:30.946229 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:30.946092 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" event={"ID":"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4","Type":"ContainerDied","Data":"f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1"} Jul 24 15:35:30.946526 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:30.946511 2557 scope.go:117] "RemoveContainer" containerID="f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1" Jul 24 15:35:30.947517 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:30.947491 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:30.947571 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:30.947555 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:30.947615 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:30.947585 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:31.953839 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:31.953810 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" event={"ID":"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4","Type":"ContainerStarted","Data":"4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5"} Jul 24 15:35:32.317689 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:32.317585 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:32.317931 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:32.317718 2557 patch_prober.go:28] interesting pod/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 15:35:32.317931 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:32.317763 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 15:35:33.317657 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:33.317614 2557 patch_prober.go:28] interesting pod/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 15:35:33.318342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:33.318303 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 15:35:34.317808 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:34.317766 2557 patch_prober.go:28] interesting pod/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 15:35:34.318232 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:34.317840 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 15:35:35.318041 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:35.317502 2557 patch_prober.go:28] interesting pod/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 15:35:35.318041 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:35.317591 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 15:35:36.320884 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.320845 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:36.321517 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.321210 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:36.322037 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.322014 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:35:36.928335 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.927143 2557 patch_prober.go:28] interesting pod/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 15:35:36.928335 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.927220 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 15:35:36.980839 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.980807 2557 generic.go:356] "Generic (PLEG): container finished" podID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerID="6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec" exitCode=0 Jul 24 15:35:36.981020 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.980879 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" event={"ID":"ecc9d46d-cd70-42d0-87d9-97b9e750375e","Type":"ContainerDied","Data":"6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec"} Jul 24 15:35:36.981460 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.981440 2557 scope.go:117] "RemoveContainer" containerID="6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec" Jul 24 15:35:36.982461 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.982433 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:36.982586 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.982516 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:36.982586 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:36.982564 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:35:37.987747 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:37.987710 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" event={"ID":"ecc9d46d-cd70-42d0-87d9-97b9e750375e","Type":"ContainerStarted","Data":"6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1"} Jul 24 15:35:38.289246 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:38.288679 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:38.289246 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:38.288801 2557 patch_prober.go:28] interesting pod/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 15:35:38.289246 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:38.288860 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 15:35:39.288448 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:39.288405 2557 patch_prober.go:28] interesting pod/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 15:35:39.289053 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:39.289021 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 15:35:40.288440 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:40.288402 2557 patch_prober.go:28] interesting pod/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 15:35:40.288784 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:40.288753 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 15:35:41.288365 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:41.288329 2557 patch_prober.go:28] interesting pod/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 15:35:41.288772 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:41.288738 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 15:35:42.291340 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:42.291309 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:42.291724 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:42.291645 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:35:42.292484 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:35:42.292462 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:36:21.777430 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:36:21.777399 2557 scope.go:117] "RemoveContainer" containerID="328f01606800e6481a873ce1f63f2c752271561d03ed7628ef1405153cc24d32" Jul 24 15:38:42.046399 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.046361 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj"] Jul 24 15:38:42.049946 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.049923 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.054011 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.053984 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d419ce\"/\"openshift-service-ca.crt\"" Jul 24 15:38:42.054134 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.054086 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d419ce\"/\"sub-test-model-d419ce-kserve-self-signed-certs\"" Jul 24 15:38:42.054214 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.054154 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d419ce\"/\"kube-root-ca.crt\"" Jul 24 15:38:42.054214 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.054172 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d419ce\"/\"default-dockercfg-zh828\"" Jul 24 15:38:42.063509 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.063483 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj"] Jul 24 15:38:42.168592 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.168553 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-model-cache\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.168770 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.168615 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-home\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.168770 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.168676 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2cvs\" (UniqueName: \"kubernetes.io/projected/04bedbbf-9166-4b70-a09a-490c94bbe91f-kube-api-access-x2cvs\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.168770 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.168707 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-tmp-dir\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.168885 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.168805 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bedbbf-9166-4b70-a09a-490c94bbe91f-tls-certs\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.168885 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.168828 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-dshm\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.270018 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.269985 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bedbbf-9166-4b70-a09a-490c94bbe91f-tls-certs\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.270018 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.270021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-dshm\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.270311 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.270047 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-model-cache\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.270311 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.270077 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-home\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.270311 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.270099 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2cvs\" (UniqueName: \"kubernetes.io/projected/04bedbbf-9166-4b70-a09a-490c94bbe91f-kube-api-access-x2cvs\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.270311 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.270116 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-tmp-dir\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.270551 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.270526 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-model-cache\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.270607 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.270566 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-tmp-dir\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.270607 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.270585 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-home\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.272460 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.272438 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-dshm\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.272611 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.272596 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bedbbf-9166-4b70-a09a-490c94bbe91f-tls-certs\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.278961 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.278944 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2cvs\" (UniqueName: \"kubernetes.io/projected/04bedbbf-9166-4b70-a09a-490c94bbe91f-kube-api-access-x2cvs\") pod \"sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.358920 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.358836 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:42.491018 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.490879 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj"] Jul 24 15:38:42.493146 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:38:42.493121 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04bedbbf_9166_4b70_a09a_490c94bbe91f.slice/crio-1b7182d0f3116f58d65a8052999581193ab29ef59fb54c22168da21bc531b3c7 WatchSource:0}: Error finding container 1b7182d0f3116f58d65a8052999581193ab29ef59fb54c22168da21bc531b3c7: Status 404 returned error can't find the container with id 1b7182d0f3116f58d65a8052999581193ab29ef59fb54c22168da21bc531b3c7 Jul 24 15:38:42.844100 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.844064 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" event={"ID":"04bedbbf-9166-4b70-a09a-490c94bbe91f","Type":"ContainerStarted","Data":"f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4"} Jul 24 15:38:42.844298 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.844114 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" event={"ID":"04bedbbf-9166-4b70-a09a-490c94bbe91f","Type":"ContainerStarted","Data":"1b7182d0f3116f58d65a8052999581193ab29ef59fb54c22168da21bc531b3c7"} Jul 24 15:38:42.862077 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:42.862028 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" podStartSLOduration=0.862013189 podStartE2EDuration="862.013189ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:38:42.859633644 +0000 UTC m=+2541.807127230" watchObservedRunningTime="2026-07-24 15:38:42.862013189 +0000 UTC m=+2541.809506775" Jul 24 15:38:52.359280 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:52.359246 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:52.359814 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:52.359383 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:52.371922 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:52.371897 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:38:52.904265 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:38:52.904234 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:40:07.418563 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:07.418527 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj"] Jul 24 15:40:22.439172 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.439124 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" podUID="04bedbbf-9166-4b70-a09a-490c94bbe91f" containerName="main" containerID="cri-o://f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4" gracePeriod=45 Jul 24 15:40:22.698479 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.698410 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:40:22.789084 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789055 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-model-cache\") pod \"04bedbbf-9166-4b70-a09a-490c94bbe91f\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " Jul 24 15:40:22.789292 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789098 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-tmp-dir\") pod \"04bedbbf-9166-4b70-a09a-490c94bbe91f\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " Jul 24 15:40:22.789292 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789129 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bedbbf-9166-4b70-a09a-490c94bbe91f-tls-certs\") pod \"04bedbbf-9166-4b70-a09a-490c94bbe91f\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " Jul 24 15:40:22.789292 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789172 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-dshm\") pod \"04bedbbf-9166-4b70-a09a-490c94bbe91f\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " Jul 24 15:40:22.789292 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789224 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-home\") pod \"04bedbbf-9166-4b70-a09a-490c94bbe91f\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " Jul 24 15:40:22.789515 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789343 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2cvs\" (UniqueName: \"kubernetes.io/projected/04bedbbf-9166-4b70-a09a-490c94bbe91f-kube-api-access-x2cvs\") pod \"04bedbbf-9166-4b70-a09a-490c94bbe91f\" (UID: \"04bedbbf-9166-4b70-a09a-490c94bbe91f\") " Jul 24 15:40:22.789515 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789416 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-model-cache" (OuterVolumeSpecName: "model-cache") pod "04bedbbf-9166-4b70-a09a-490c94bbe91f" (UID: "04bedbbf-9166-4b70-a09a-490c94bbe91f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:40:22.789515 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789434 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04bedbbf-9166-4b70-a09a-490c94bbe91f" (UID: "04bedbbf-9166-4b70-a09a-490c94bbe91f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:40:22.789663 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789559 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-home" (OuterVolumeSpecName: "home") pod "04bedbbf-9166-4b70-a09a-490c94bbe91f" (UID: "04bedbbf-9166-4b70-a09a-490c94bbe91f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:40:22.789768 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789721 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:40:22.789768 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789739 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:40:22.789768 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.789754 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:40:22.792043 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.792020 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-dshm" (OuterVolumeSpecName: "dshm") pod "04bedbbf-9166-4b70-a09a-490c94bbe91f" (UID: "04bedbbf-9166-4b70-a09a-490c94bbe91f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:40:22.792221 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.792139 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bedbbf-9166-4b70-a09a-490c94bbe91f-kube-api-access-x2cvs" (OuterVolumeSpecName: "kube-api-access-x2cvs") pod "04bedbbf-9166-4b70-a09a-490c94bbe91f" (UID: "04bedbbf-9166-4b70-a09a-490c94bbe91f"). InnerVolumeSpecName "kube-api-access-x2cvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:40:22.792429 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.792402 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bedbbf-9166-4b70-a09a-490c94bbe91f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04bedbbf-9166-4b70-a09a-490c94bbe91f" (UID: "04bedbbf-9166-4b70-a09a-490c94bbe91f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:40:22.890487 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.890453 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bedbbf-9166-4b70-a09a-490c94bbe91f-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:40:22.890487 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.890483 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bedbbf-9166-4b70-a09a-490c94bbe91f-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:40:22.890487 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:22.890492 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2cvs\" (UniqueName: \"kubernetes.io/projected/04bedbbf-9166-4b70-a09a-490c94bbe91f-kube-api-access-x2cvs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:40:23.318481 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.318443 2557 generic.go:356] "Generic (PLEG): container finished" podID="04bedbbf-9166-4b70-a09a-490c94bbe91f" containerID="f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4" exitCode=0 Jul 24 15:40:23.318695 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.318494 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" Jul 24 15:40:23.318695 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.318486 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" event={"ID":"04bedbbf-9166-4b70-a09a-490c94bbe91f","Type":"ContainerDied","Data":"f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4"} Jul 24 15:40:23.318695 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.318618 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj" event={"ID":"04bedbbf-9166-4b70-a09a-490c94bbe91f","Type":"ContainerDied","Data":"1b7182d0f3116f58d65a8052999581193ab29ef59fb54c22168da21bc531b3c7"} Jul 24 15:40:23.318695 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.318642 2557 scope.go:117] "RemoveContainer" containerID="f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4" Jul 24 15:40:23.328804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.328786 2557 scope.go:117] "RemoveContainer" containerID="f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4" Jul 24 15:40:23.329103 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:40:23.329079 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4\": container with ID starting with f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4 not found: ID does not exist" containerID="f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4" Jul 24 15:40:23.329175 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.329117 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4"} err="failed to get container status \"f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4\": rpc error: code = NotFound desc = could not find container \"f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4\": container with ID starting with f77011828cc4ae58172190cd92e1d817af56f0da300d41cb18f7b102c51ac0e4 not found: ID does not exist" Jul 24 15:40:23.340899 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.340864 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj"] Jul 24 15:40:23.344324 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.344296 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d419ce/sub-test-model-d419ce-kserve-6d4b579ddc-fc7kj"] Jul 24 15:40:23.588950 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:40:23.588868 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04bedbbf-9166-4b70-a09a-490c94bbe91f" path="/var/lib/kubelet/pods/04bedbbf-9166-4b70-a09a-490c94bbe91f/volumes" Jul 24 15:41:59.866308 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.866275 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4"] Jul 24 15:41:59.867156 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.866775 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bedbbf-9166-4b70-a09a-490c94bbe91f" containerName="main" Jul 24 15:41:59.867156 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.866796 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bedbbf-9166-4b70-a09a-490c94bbe91f" containerName="main" Jul 24 15:41:59.867156 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.866926 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bedbbf-9166-4b70-a09a-490c94bbe91f" containerName="main" Jul 24 15:41:59.870002 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.869981 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:41:59.872713 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.872690 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-cd9c98-openshift-default-dockercfg-mpwng\"" Jul 24 15:41:59.883771 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.883743 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4"] Jul 24 15:41:59.910457 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.910424 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-credential-socket\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:41:59.910457 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.910459 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9gdr\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-kube-api-access-z9gdr\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:41:59.910709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.910499 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-socket\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:41:59.910709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.910568 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-certs\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:41:59.910709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.910616 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-token\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:41:59.910709 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.910684 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-data\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:41:59.910905 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.910744 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-podinfo\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:41:59.910905 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.910775 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-envoy\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:41:59.910905 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:41:59.910812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b553265-a5e0-4d19-a852-98a5a86f4a70-istiod-ca-cert\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012324 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012283 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-envoy\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012324 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012326 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b553265-a5e0-4d19-a852-98a5a86f4a70-istiod-ca-cert\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012476 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-credential-socket\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012508 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9gdr\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-kube-api-access-z9gdr\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012536 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-socket\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012693 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012661 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-certs\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012750 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012716 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-token\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012785 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-data\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012804 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012789 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-credential-socket\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012904 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012862 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-podinfo\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.012963 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.012896 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-socket\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.013019 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.013004 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-certs\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.013095 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.013064 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-data\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.013095 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.013074 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b553265-a5e0-4d19-a852-98a5a86f4a70-istiod-ca-cert\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.015622 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.015598 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-envoy\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.015930 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.015909 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-podinfo\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.022623 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.022596 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-token\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.022737 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.022653 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9gdr\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-kube-api-access-z9gdr\") pod \"e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.180132 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.180086 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:00.323557 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.323499 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4"] Jul 24 15:42:00.325840 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:42:00.325811 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b553265_a5e0_4d19_a852_98a5a86f4a70.slice/crio-7138dc52399cf03362348bd5ddb7ff533f0fd377f341d53d8aba73c08216abd7 WatchSource:0}: Error finding container 7138dc52399cf03362348bd5ddb7ff533f0fd377f341d53d8aba73c08216abd7: Status 404 returned error can't find the container with id 7138dc52399cf03362348bd5ddb7ff533f0fd377f341d53d8aba73c08216abd7 Jul 24 15:42:00.327986 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.327956 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:42:00.328055 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.328022 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:42:00.328055 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.328049 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:42:00.771233 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.771164 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" event={"ID":"7b553265-a5e0-4d19-a852-98a5a86f4a70","Type":"ContainerStarted","Data":"949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1"} Jul 24 15:42:00.771391 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.771294 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" event={"ID":"7b553265-a5e0-4d19-a852-98a5a86f4a70","Type":"ContainerStarted","Data":"7138dc52399cf03362348bd5ddb7ff533f0fd377f341d53d8aba73c08216abd7"} Jul 24 15:42:00.789554 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:00.789494 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" podStartSLOduration=1.789475124 podStartE2EDuration="1.789475124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:00.789296839 +0000 UTC m=+2739.736790445" watchObservedRunningTime="2026-07-24 15:42:00.789475124 +0000 UTC m=+2739.736968711" Jul 24 15:42:01.180926 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:01.180889 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:01.186007 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:01.185982 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:01.776148 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:01.776116 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:01.777073 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:01.777054 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:03.569002 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.568969 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v"] Jul 24 15:42:03.572969 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.572946 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:03.575869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.575845 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-cd9c98-serving-cert\"" Jul 24 15:42:03.589109 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.589078 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v"] Jul 24 15:42:03.646408 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.646367 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz7dg\" (UniqueName: \"kubernetes.io/projected/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-kube-api-access-cz7dg\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:03.646596 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.646424 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:03.646596 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.646533 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:03.747891 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.747854 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7dg\" (UniqueName: \"kubernetes.io/projected/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-kube-api-access-cz7dg\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:03.748080 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.747904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:03.748080 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.747952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:03.748080 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:42:03.748059 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-serving-cert: secret "maas-api-e2e-ipp-a-cd9c98-serving-cert" not found Jul 24 15:42:03.748240 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:42:03.748140 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-maas-api-tls podName:3f9bbfb7-6012-4d18-958b-f79d2fa451c7 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:04.248116677 +0000 UTC m=+2743.195610240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-maas-api-tls") pod "maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" (UID: "3f9bbfb7-6012-4d18-958b-f79d2fa451c7") : secret "maas-api-e2e-ipp-a-cd9c98-serving-cert" not found Jul 24 15:42:03.748596 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.748578 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:03.757305 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:03.757276 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7dg\" (UniqueName: \"kubernetes.io/projected/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-kube-api-access-cz7dg\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:04.253398 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:04.253357 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:04.256202 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:04.256150 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:04.486660 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:04.486624 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:04.626632 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:04.626605 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v"] Jul 24 15:42:04.628544 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:42:04.628520 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9bbfb7_6012_4d18_958b_f79d2fa451c7.slice/crio-5419a8980f2ff97fbaa7f51b371fce83121ca4e350cacf254e6f803a8ce33eb8 WatchSource:0}: Error finding container 5419a8980f2ff97fbaa7f51b371fce83121ca4e350cacf254e6f803a8ce33eb8: Status 404 returned error can't find the container with id 5419a8980f2ff97fbaa7f51b371fce83121ca4e350cacf254e6f803a8ce33eb8 Jul 24 15:42:04.630429 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:04.630412 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:42:04.791614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:04.791526 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" event={"ID":"3f9bbfb7-6012-4d18-958b-f79d2fa451c7","Type":"ContainerStarted","Data":"5419a8980f2ff97fbaa7f51b371fce83121ca4e350cacf254e6f803a8ce33eb8"} Jul 24 15:42:05.797604 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:05.797560 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" event={"ID":"3f9bbfb7-6012-4d18-958b-f79d2fa451c7","Type":"ContainerStarted","Data":"141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627"} Jul 24 15:42:05.797998 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:05.797736 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:05.816811 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:05.816757 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" podStartSLOduration=2.3030269150000002 podStartE2EDuration="2.816743256s" totalImagesPullingTime="513.716341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:05.814403259 +0000 UTC m=+2744.761896845" watchObservedRunningTime="2026-07-24 15:42:05.816743256 +0000 UTC m=+2744.764236842" Jul 24 15:42:11.809932 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:11.809892 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:42:17.817414 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:17.817364 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 15:42:17.817887 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:17.817496 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 15:42:19.867696 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:19.867581 2557 generic.go:356] "Generic (PLEG): container finished" podID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerID="949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1" exitCode=0 Jul 24 15:42:19.867696 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:19.867661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" event={"ID":"7b553265-a5e0-4d19-a852-98a5a86f4a70","Type":"ContainerDied","Data":"949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1"} Jul 24 15:42:19.868259 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:19.868089 2557 scope.go:117] "RemoveContainer" containerID="949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1" Jul 24 15:42:19.869167 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:19.869137 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:42:19.869300 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:19.869224 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:42:19.869300 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:19.869257 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:42:20.874384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:20.874349 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" event={"ID":"7b553265-a5e0-4d19-a852-98a5a86f4a70","Type":"ContainerStarted","Data":"213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf"} Jul 24 15:42:21.180617 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:21.180579 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:21.180905 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:21.180878 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 24 15:42:21.180963 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:21.180936 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 24 15:42:22.180594 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:22.180559 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 24 15:42:22.180983 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:22.180611 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 24 15:42:23.180952 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:23.180914 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 24 15:42:23.181374 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:23.180971 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 24 15:42:24.180897 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:24.180859 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 24 15:42:24.181081 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:24.180928 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 24 15:42:25.184377 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:25.184344 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:25.184852 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:25.184732 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:42:25.185407 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:42:25.185379 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:43:36.766455 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.766420 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk"] Jul 24 15:43:36.769879 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.769859 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.772318 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.772297 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-923368-openshift-default-dockercfg-2wkpt\"" Jul 24 15:43:36.783322 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.783291 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk"] Jul 24 15:43:36.827385 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.827346 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-podinfo\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.827385 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.827388 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-credential-socket\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.827639 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.827409 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-envoy\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.827639 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.827537 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-certs\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.827639 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.827574 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-socket\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.827755 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.827681 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-token\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.827755 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.827715 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-data\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.827755 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.827742 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a17adc-861c-421c-ae72-28a9b4f8813c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.827869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.827767 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zc9z\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-kube-api-access-4zc9z\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928282 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928242 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-token\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928282 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928288 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-data\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928542 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928308 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a17adc-861c-421c-ae72-28a9b4f8813c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928542 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928328 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zc9z\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-kube-api-access-4zc9z\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928542 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928371 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-podinfo\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928542 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-credential-socket\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928542 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928412 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-envoy\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928542 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928475 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-certs\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928542 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928507 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-socket\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928915 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928730 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-data\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928915 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-credential-socket\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.928915 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.928859 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-socket\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.929081 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.929050 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-certs\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.929132 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.929107 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a17adc-861c-421c-ae72-28a9b4f8813c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.930898 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.930875 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-envoy\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.931031 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.931016 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-podinfo\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.936084 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.936055 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-token\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:36.936259 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:36.936242 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zc9z\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-kube-api-access-4zc9z\") pod \"e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:37.079932 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:37.079842 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:37.226292 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:37.226263 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk"] Jul 24 15:43:37.228370 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:43:37.228337 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a17adc_861c_421c_ae72_28a9b4f8813c.slice/crio-75b68095ba935db2933b9204a8903432472d5748be71aeba7b1ea64f0d8fcae4 WatchSource:0}: Error finding container 75b68095ba935db2933b9204a8903432472d5748be71aeba7b1ea64f0d8fcae4: Status 404 returned error can't find the container with id 75b68095ba935db2933b9204a8903432472d5748be71aeba7b1ea64f0d8fcae4 Jul 24 15:43:37.230629 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:37.230594 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:43:37.230724 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:37.230662 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:43:37.230724 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:37.230693 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:43:37.242072 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:37.242046 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" event={"ID":"d1a17adc-861c-421c-ae72-28a9b4f8813c","Type":"ContainerStarted","Data":"75b68095ba935db2933b9204a8903432472d5748be71aeba7b1ea64f0d8fcae4"} Jul 24 15:43:38.248349 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:38.248310 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" event={"ID":"d1a17adc-861c-421c-ae72-28a9b4f8813c","Type":"ContainerStarted","Data":"169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99"} Jul 24 15:43:38.271241 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:38.271172 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" podStartSLOduration=2.271156413 podStartE2EDuration="2.271156413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:43:38.269259543 +0000 UTC m=+2837.216753149" watchObservedRunningTime="2026-07-24 15:43:38.271156413 +0000 UTC m=+2837.218650003" Jul 24 15:43:39.080727 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:39.080687 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:39.086223 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:39.086167 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:39.253241 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:39.253207 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:43:39.254118 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:43:39.254101 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:44:00.353128 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:00.353096 2557 generic.go:356] "Generic (PLEG): container finished" podID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerID="169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99" exitCode=0 Jul 24 15:44:00.353571 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:00.353204 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" event={"ID":"d1a17adc-861c-421c-ae72-28a9b4f8813c","Type":"ContainerDied","Data":"169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99"} Jul 24 15:44:00.353654 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:00.353639 2557 scope.go:117] "RemoveContainer" containerID="169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99" Jul 24 15:44:00.354728 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:00.354686 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:44:00.354813 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:00.354796 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:44:00.354853 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:00.354834 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 15:44:01.361435 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:01.361401 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" event={"ID":"d1a17adc-861c-421c-ae72-28a9b4f8813c","Type":"ContainerStarted","Data":"0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469"} Jul 24 15:44:02.080511 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:02.080477 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:44:02.080693 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:02.080676 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 24 15:44:02.080758 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:02.080736 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 24 15:44:03.080465 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:03.080427 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 24 15:44:03.080886 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:03.080515 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 24 15:44:04.080419 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:04.080385 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 24 15:44:04.080605 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:04.080474 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 24 15:44:05.081064 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:05.081033 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 24 15:44:05.081556 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:05.081089 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 24 15:44:06.085121 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:06.085091 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:44:06.085553 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:06.085394 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:44:06.086108 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:06.086091 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:44:18.083689 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:18.083605 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk"] Jul 24 15:44:18.084246 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:18.083864 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" containerID="cri-o://0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469" gracePeriod=30 Jul 24 15:44:21.088012 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:21.087975 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:44:21.088398 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:21.088041 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:44:23.331627 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.331603 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:44:23.479457 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.479424 2557 generic.go:356] "Generic (PLEG): container finished" podID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerID="0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469" exitCode=0 Jul 24 15:44:23.479648 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.479476 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" Jul 24 15:44:23.479648 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.479493 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" event={"ID":"d1a17adc-861c-421c-ae72-28a9b4f8813c","Type":"ContainerDied","Data":"0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469"} Jul 24 15:44:23.479648 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.479538 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk" event={"ID":"d1a17adc-861c-421c-ae72-28a9b4f8813c","Type":"ContainerDied","Data":"75b68095ba935db2933b9204a8903432472d5748be71aeba7b1ea64f0d8fcae4"} Jul 24 15:44:23.479648 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.479560 2557 scope.go:117] "RemoveContainer" containerID="0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469" Jul 24 15:44:23.488550 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.488527 2557 scope.go:117] "RemoveContainer" containerID="169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99" Jul 24 15:44:23.494793 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.494771 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-token\") pod \"d1a17adc-861c-421c-ae72-28a9b4f8813c\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " Jul 24 15:44:23.494906 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.494804 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-credential-socket\") pod \"d1a17adc-861c-421c-ae72-28a9b4f8813c\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " Jul 24 15:44:23.494906 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.494845 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-data\") pod \"d1a17adc-861c-421c-ae72-28a9b4f8813c\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " Jul 24 15:44:23.494906 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.494863 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zc9z\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-kube-api-access-4zc9z\") pod \"d1a17adc-861c-421c-ae72-28a9b4f8813c\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " Jul 24 15:44:23.494906 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.494897 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a17adc-861c-421c-ae72-28a9b4f8813c-istiod-ca-cert\") pod \"d1a17adc-861c-421c-ae72-28a9b4f8813c\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " Jul 24 15:44:23.495109 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.494959 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-podinfo\") pod \"d1a17adc-861c-421c-ae72-28a9b4f8813c\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " Jul 24 15:44:23.495201 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.495156 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d1a17adc-861c-421c-ae72-28a9b4f8813c" (UID: "d1a17adc-861c-421c-ae72-28a9b4f8813c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:44:23.495384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.495354 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a17adc-861c-421c-ae72-28a9b4f8813c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d1a17adc-861c-421c-ae72-28a9b4f8813c" (UID: "d1a17adc-861c-421c-ae72-28a9b4f8813c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:44:23.495384 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.495380 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-certs\") pod \"d1a17adc-861c-421c-ae72-28a9b4f8813c\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " Jul 24 15:44:23.495734 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.495427 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-envoy\") pod \"d1a17adc-861c-421c-ae72-28a9b4f8813c\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " Jul 24 15:44:23.495734 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.495469 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-socket\") pod \"d1a17adc-861c-421c-ae72-28a9b4f8813c\" (UID: \"d1a17adc-861c-421c-ae72-28a9b4f8813c\") " Jul 24 15:44:23.495734 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.495617 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d1a17adc-861c-421c-ae72-28a9b4f8813c" (UID: "d1a17adc-861c-421c-ae72-28a9b4f8813c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:44:23.495909 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.495879 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d1a17adc-861c-421c-ae72-28a9b4f8813c" (UID: "d1a17adc-861c-421c-ae72-28a9b4f8813c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:44:23.496045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.495978 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:23.496045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.496002 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a17adc-861c-421c-ae72-28a9b4f8813c-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:23.496045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.496015 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:23.496045 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.496027 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:23.496318 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.496057 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d1a17adc-861c-421c-ae72-28a9b4f8813c" (UID: "d1a17adc-861c-421c-ae72-28a9b4f8813c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:44:23.497442 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.497418 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d1a17adc-861c-421c-ae72-28a9b4f8813c" (UID: "d1a17adc-861c-421c-ae72-28a9b4f8813c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:44:23.497535 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.497461 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-kube-api-access-4zc9z" (OuterVolumeSpecName: "kube-api-access-4zc9z") pod "d1a17adc-861c-421c-ae72-28a9b4f8813c" (UID: "d1a17adc-861c-421c-ae72-28a9b4f8813c"). InnerVolumeSpecName "kube-api-access-4zc9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:44:23.497736 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.497711 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d1a17adc-861c-421c-ae72-28a9b4f8813c" (UID: "d1a17adc-861c-421c-ae72-28a9b4f8813c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:44:23.497821 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.497803 2557 scope.go:117] "RemoveContainer" containerID="0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469" Jul 24 15:44:23.497956 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.497933 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d1a17adc-861c-421c-ae72-28a9b4f8813c" (UID: "d1a17adc-861c-421c-ae72-28a9b4f8813c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:44:23.498120 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:44:23.498100 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469\": container with ID starting with 0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469 not found: ID does not exist" containerID="0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469" Jul 24 15:44:23.498177 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.498130 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469"} err="failed to get container status \"0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469\": rpc error: code = NotFound desc = could not find container \"0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469\": container with ID starting with 0655b7358a9b68284fbed7efcd5ead0921a0c7933fdb5fa30eae362fc0891469 not found: ID does not exist" Jul 24 15:44:23.498177 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.498150 2557 scope.go:117] "RemoveContainer" containerID="169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99" Jul 24 15:44:23.498408 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:44:23.498390 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99\": container with ID starting with 169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99 not found: ID does not exist" containerID="169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99" Jul 24 15:44:23.498465 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.498413 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99"} err="failed to get container status \"169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99\": rpc error: code = NotFound desc = could not find container \"169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99\": container with ID starting with 169ec670d5a3daa2c7123d4d8a2aed8df8c6610206cb7debc0fac2aec7d30e99 not found: ID does not exist" Jul 24 15:44:23.596617 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.596594 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:23.596617 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.596617 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:23.596754 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.596627 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zc9z\" (UniqueName: \"kubernetes.io/projected/d1a17adc-861c-421c-ae72-28a9b4f8813c-kube-api-access-4zc9z\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:23.596754 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.596637 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:23.596754 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.596645 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a17adc-861c-421c-ae72-28a9b4f8813c-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:23.796395 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.796319 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk"] Jul 24 15:44:23.798030 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:23.798007 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-923368-openshift-default-7886d898cd-kgzzk"] Jul 24 15:44:25.588112 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:25.588077 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" path="/var/lib/kubelet/pods/d1a17adc-861c-421c-ae72-28a9b4f8813c/volumes" Jul 24 15:44:42.086330 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.086299 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v"] Jul 24 15:44:42.086752 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.086501 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" podUID="3f9bbfb7-6012-4d18-958b-f79d2fa451c7" containerName="maas-api" containerID="cri-o://141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627" gracePeriod=30 Jul 24 15:44:42.367601 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.367573 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:44:42.477085 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.477054 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-maas-api-tls\") pod \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " Jul 24 15:44:42.477280 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.477098 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz7dg\" (UniqueName: \"kubernetes.io/projected/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-kube-api-access-cz7dg\") pod \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " Jul 24 15:44:42.477280 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.477154 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-openshift-service-ca\") pod \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\" (UID: \"3f9bbfb7-6012-4d18-958b-f79d2fa451c7\") " Jul 24 15:44:42.477662 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.477631 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f9bbfb7-6012-4d18-958b-f79d2fa451c7" (UID: "3f9bbfb7-6012-4d18-958b-f79d2fa451c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:44:42.479259 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.479236 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f9bbfb7-6012-4d18-958b-f79d2fa451c7" (UID: "3f9bbfb7-6012-4d18-958b-f79d2fa451c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:44:42.479357 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.479289 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-kube-api-access-cz7dg" (OuterVolumeSpecName: "kube-api-access-cz7dg") pod "3f9bbfb7-6012-4d18-958b-f79d2fa451c7" (UID: "3f9bbfb7-6012-4d18-958b-f79d2fa451c7"). InnerVolumeSpecName "kube-api-access-cz7dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:44:42.570467 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.570429 2557 generic.go:356] "Generic (PLEG): container finished" podID="3f9bbfb7-6012-4d18-958b-f79d2fa451c7" containerID="141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627" exitCode=0 Jul 24 15:44:42.570640 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.570478 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" Jul 24 15:44:42.577364 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.570515 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" event={"ID":"3f9bbfb7-6012-4d18-958b-f79d2fa451c7","Type":"ContainerDied","Data":"141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627"} Jul 24 15:44:42.577364 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.571087 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v" event={"ID":"3f9bbfb7-6012-4d18-958b-f79d2fa451c7","Type":"ContainerDied","Data":"5419a8980f2ff97fbaa7f51b371fce83121ca4e350cacf254e6f803a8ce33eb8"} Jul 24 15:44:42.577364 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.571134 2557 scope.go:117] "RemoveContainer" containerID="141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627" Jul 24 15:44:42.578621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.578464 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:42.578621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.578484 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:42.578621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.578493 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz7dg\" (UniqueName: \"kubernetes.io/projected/3f9bbfb7-6012-4d18-958b-f79d2fa451c7-kube-api-access-cz7dg\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:44:42.586482 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.586462 2557 scope.go:117] "RemoveContainer" containerID="141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627" Jul 24 15:44:42.586711 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:44:42.586695 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627\": container with ID starting with 141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627 not found: ID does not exist" containerID="141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627" Jul 24 15:44:42.586752 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.586719 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627"} err="failed to get container status \"141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627\": rpc error: code = NotFound desc = could not find container \"141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627\": container with ID starting with 141a1e4c9e839904e7a4e3581b7c04beeab07bab69a2c49238e38265a2955627 not found: ID does not exist" Jul 24 15:44:42.595569 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.595539 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v"] Jul 24 15:44:42.600750 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:42.600700 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cd9c98-58887848dc-4nl5v"] Jul 24 15:44:43.589426 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:44:43.589382 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f9bbfb7-6012-4d18-958b-f79d2fa451c7" path="/var/lib/kubelet/pods/3f9bbfb7-6012-4d18-958b-f79d2fa451c7/volumes" Jul 24 15:45:00.143060 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143021 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb"] Jul 24 15:45:00.143545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143375 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" Jul 24 15:45:00.143545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143386 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" Jul 24 15:45:00.143545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143401 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" Jul 24 15:45:00.143545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143406 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" Jul 24 15:45:00.143545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143415 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9bbfb7-6012-4d18-958b-f79d2fa451c7" containerName="maas-api" Jul 24 15:45:00.143545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143421 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9bbfb7-6012-4d18-958b-f79d2fa451c7" containerName="maas-api" Jul 24 15:45:00.143545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143482 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9bbfb7-6012-4d18-958b-f79d2fa451c7" containerName="maas-api" Jul 24 15:45:00.143545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143489 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" Jul 24 15:45:00.143545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.143495 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a17adc-861c-421c-ae72-28a9b4f8813c" containerName="istio-proxy" Jul 24 15:45:00.146382 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.146363 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb" Jul 24 15:45:00.148842 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.148824 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:45:00.156272 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.156247 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb"] Jul 24 15:45:00.158692 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.158673 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb" Jul 24 15:45:00.294929 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.294907 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb"] Jul 24 15:45:00.297044 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:45:00.297012 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d7a321_59ed_4af3_9374_ca9e4147869e.slice/crio-de935e9d4329a7943f728a36efdaa01545adc169c0982dbf730bbe1d1232b847 WatchSource:0}: Error finding container de935e9d4329a7943f728a36efdaa01545adc169c0982dbf730bbe1d1232b847: Status 404 returned error can't find the container with id de935e9d4329a7943f728a36efdaa01545adc169c0982dbf730bbe1d1232b847 Jul 24 15:45:00.658325 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.658282 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb" event={"ID":"21d7a321-59ed-4af3-9374-ca9e4147869e","Type":"ContainerStarted","Data":"a21f2b611eac930e30e73e5be19a7a589770850d0972d1b5ccf5ca6b2d0d1f8c"} Jul 24 15:45:00.658504 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.658353 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb" event={"ID":"21d7a321-59ed-4af3-9374-ca9e4147869e","Type":"ContainerStarted","Data":"de935e9d4329a7943f728a36efdaa01545adc169c0982dbf730bbe1d1232b847"} Jul 24 15:45:00.675008 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:00.674958 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb" podStartSLOduration=0.67494499 podStartE2EDuration="674.94499ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:00.67317807 +0000 UTC m=+2919.620671657" watchObservedRunningTime="2026-07-24 15:45:00.67494499 +0000 UTC m=+2919.622438576" Jul 24 15:45:01.664685 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:01.664653 2557 generic.go:356] "Generic (PLEG): container finished" podID="21d7a321-59ed-4af3-9374-ca9e4147869e" containerID="a21f2b611eac930e30e73e5be19a7a589770850d0972d1b5ccf5ca6b2d0d1f8c" exitCode=0 Jul 24 15:45:01.665079 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:01.664735 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb" event={"ID":"21d7a321-59ed-4af3-9374-ca9e4147869e","Type":"ContainerDied","Data":"a21f2b611eac930e30e73e5be19a7a589770850d0972d1b5ccf5ca6b2d0d1f8c"} Jul 24 15:45:02.796781 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:02.796754 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb" Jul 24 15:45:03.674789 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:03.674765 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb" Jul 24 15:45:03.674958 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:03.674806 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-hvdkb" event={"ID":"21d7a321-59ed-4af3-9374-ca9e4147869e","Type":"ContainerDied","Data":"de935e9d4329a7943f728a36efdaa01545adc169c0982dbf730bbe1d1232b847"} Jul 24 15:45:03.674958 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:03.674849 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de935e9d4329a7943f728a36efdaa01545adc169c0982dbf730bbe1d1232b847" Jul 24 15:45:50.825263 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:50.825162 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4"] Jul 24 15:45:50.825756 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:50.825477 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" containerID="cri-o://213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf" gracePeriod=30 Jul 24 15:45:55.187067 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:55.187027 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:45:55.187471 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:55.187090 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:45:56.087966 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.087906 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:45:56.188531 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188494 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-socket\") pod \"7b553265-a5e0-4d19-a852-98a5a86f4a70\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " Jul 24 15:45:56.188531 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188531 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-token\") pod \"7b553265-a5e0-4d19-a852-98a5a86f4a70\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188553 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b553265-a5e0-4d19-a852-98a5a86f4a70-istiod-ca-cert\") pod \"7b553265-a5e0-4d19-a852-98a5a86f4a70\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188614 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-data\") pod \"7b553265-a5e0-4d19-a852-98a5a86f4a70\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188666 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-credential-socket\") pod \"7b553265-a5e0-4d19-a852-98a5a86f4a70\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188696 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9gdr\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-kube-api-access-z9gdr\") pod \"7b553265-a5e0-4d19-a852-98a5a86f4a70\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188746 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-envoy\") pod \"7b553265-a5e0-4d19-a852-98a5a86f4a70\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188783 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-certs\") pod \"7b553265-a5e0-4d19-a852-98a5a86f4a70\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188810 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b553265-a5e0-4d19-a852-98a5a86f4a70" (UID: "7b553265-a5e0-4d19-a852-98a5a86f4a70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.188835 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-podinfo\") pod \"7b553265-a5e0-4d19-a852-98a5a86f4a70\" (UID: \"7b553265-a5e0-4d19-a852-98a5a86f4a70\") " Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.189009 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b553265-a5e0-4d19-a852-98a5a86f4a70" (UID: "7b553265-a5e0-4d19-a852-98a5a86f4a70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:45:56.189054 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.189042 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b553265-a5e0-4d19-a852-98a5a86f4a70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b553265-a5e0-4d19-a852-98a5a86f4a70" (UID: "7b553265-a5e0-4d19-a852-98a5a86f4a70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:45:56.189556 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.189237 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:45:56.189556 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.189256 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b553265-a5e0-4d19-a852-98a5a86f4a70-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:45:56.189556 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.189275 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:45:56.189556 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.189336 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b553265-a5e0-4d19-a852-98a5a86f4a70" (UID: "7b553265-a5e0-4d19-a852-98a5a86f4a70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:45:56.189822 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.189797 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b553265-a5e0-4d19-a852-98a5a86f4a70" (UID: "7b553265-a5e0-4d19-a852-98a5a86f4a70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:45:56.191073 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.191052 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b553265-a5e0-4d19-a852-98a5a86f4a70" (UID: "7b553265-a5e0-4d19-a852-98a5a86f4a70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:45:56.191640 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.191618 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b553265-a5e0-4d19-a852-98a5a86f4a70" (UID: "7b553265-a5e0-4d19-a852-98a5a86f4a70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:45:56.191728 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.191685 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b553265-a5e0-4d19-a852-98a5a86f4a70" (UID: "7b553265-a5e0-4d19-a852-98a5a86f4a70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:45:56.191728 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.191697 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-kube-api-access-z9gdr" (OuterVolumeSpecName: "kube-api-access-z9gdr") pod "7b553265-a5e0-4d19-a852-98a5a86f4a70" (UID: "7b553265-a5e0-4d19-a852-98a5a86f4a70"). InnerVolumeSpecName "kube-api-access-z9gdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:45:56.290609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.290574 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:45:56.290609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.290600 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9gdr\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-kube-api-access-z9gdr\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:45:56.290609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.290610 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:45:56.290609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.290618 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:45:56.290878 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.290626 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:45:56.290878 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.290634 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b553265-a5e0-4d19-a852-98a5a86f4a70-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:45:56.930538 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.930500 2557 generic.go:356] "Generic (PLEG): container finished" podID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerID="213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf" exitCode=0 Jul 24 15:45:56.930740 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.930555 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" Jul 24 15:45:56.930740 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.930583 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" event={"ID":"7b553265-a5e0-4d19-a852-98a5a86f4a70","Type":"ContainerDied","Data":"213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf"} Jul 24 15:45:56.930740 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.930633 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4" event={"ID":"7b553265-a5e0-4d19-a852-98a5a86f4a70","Type":"ContainerDied","Data":"7138dc52399cf03362348bd5ddb7ff533f0fd377f341d53d8aba73c08216abd7"} Jul 24 15:45:56.930740 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.930655 2557 scope.go:117] "RemoveContainer" containerID="213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf" Jul 24 15:45:56.940427 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.940412 2557 scope.go:117] "RemoveContainer" containerID="949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1" Jul 24 15:45:56.948690 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.948511 2557 scope.go:117] "RemoveContainer" containerID="213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf" Jul 24 15:45:56.948751 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:45:56.948736 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf\": container with ID starting with 213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf not found: ID does not exist" containerID="213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf" Jul 24 15:45:56.948798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.948761 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf"} err="failed to get container status \"213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf\": rpc error: code = NotFound desc = could not find container \"213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf\": container with ID starting with 213906e19ecf4a7ef3e545215f8de010e204701862abdf563a7917b70d91a6bf not found: ID does not exist" Jul 24 15:45:56.948798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.948777 2557 scope.go:117] "RemoveContainer" containerID="949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1" Jul 24 15:45:56.949031 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:45:56.949016 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1\": container with ID starting with 949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1 not found: ID does not exist" containerID="949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1" Jul 24 15:45:56.949069 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.949034 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1"} err="failed to get container status \"949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1\": rpc error: code = NotFound desc = could not find container \"949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1\": container with ID starting with 949cd352b15b9de31a66c829d0294a7a7f4859ad0adb9268edbb1afb4a495da1 not found: ID does not exist" Jul 24 15:45:56.955870 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.955846 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4"] Jul 24 15:45:56.960290 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:56.960269 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-cd9c98-openshift-default-58996dc654-nzfh4"] Jul 24 15:45:57.587155 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:45:57.587125 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" path="/var/lib/kubelet/pods/7b553265-a5e0-4d19-a852-98a5a86f4a70/volumes" Jul 24 15:46:13.509094 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509053 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz"] Jul 24 15:46:13.509614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509437 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" Jul 24 15:46:13.509614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509451 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" Jul 24 15:46:13.509614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509470 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" Jul 24 15:46:13.509614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509475 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" Jul 24 15:46:13.509614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509485 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d7a321-59ed-4af3-9374-ca9e4147869e" containerName="cleanup" Jul 24 15:46:13.509614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509492 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d7a321-59ed-4af3-9374-ca9e4147869e" containerName="cleanup" Jul 24 15:46:13.509614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509550 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d7a321-59ed-4af3-9374-ca9e4147869e" containerName="cleanup" Jul 24 15:46:13.509614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509561 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" Jul 24 15:46:13.509614 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.509568 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b553265-a5e0-4d19-a852-98a5a86f4a70" containerName="istio-proxy" Jul 24 15:46:13.512398 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.512379 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" Jul 24 15:46:13.514851 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.514832 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8grb\"" Jul 24 15:46:13.519880 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.519857 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz"] Jul 24 15:46:13.655644 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.655606 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz\" (UID: \"3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" Jul 24 15:46:13.756305 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.756258 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz\" (UID: \"3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" Jul 24 15:46:13.756871 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.756850 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz\" (UID: \"3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" Jul 24 15:46:13.824332 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.824253 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" Jul 24 15:46:13.961500 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:13.961476 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz"] Jul 24 15:46:13.963735 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:46:13.963692 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a69c1d2_ce6d_4a6c_b5a6_56315b7081a7.slice/crio-1368489c8ea989f21ba1b72824c5b1b0c961f5cd53d1f969a46e4e2b1946b47b WatchSource:0}: Error finding container 1368489c8ea989f21ba1b72824c5b1b0c961f5cd53d1f969a46e4e2b1946b47b: Status 404 returned error can't find the container with id 1368489c8ea989f21ba1b72824c5b1b0c961f5cd53d1f969a46e4e2b1946b47b Jul 24 15:46:14.020037 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:14.020004 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" event={"ID":"3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7","Type":"ContainerStarted","Data":"1368489c8ea989f21ba1b72824c5b1b0c961f5cd53d1f969a46e4e2b1946b47b"} Jul 24 15:46:15.026080 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:15.025993 2557 generic.go:356] "Generic (PLEG): container finished" podID="3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7" containerID="de3770fc17f87318f5e3568999ed62a971ab34ecb837047f7e6eccac697a3b4a" exitCode=0 Jul 24 15:46:15.026080 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:15.026042 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" event={"ID":"3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7","Type":"ContainerDied","Data":"de3770fc17f87318f5e3568999ed62a971ab34ecb837047f7e6eccac697a3b4a"} Jul 24 15:46:16.168928 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:16.168904 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" Jul 24 15:46:16.286932 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:16.286849 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7-maas-api-service-ca\") pod \"3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7\" (UID: \"3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7\") " Jul 24 15:46:16.287276 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:16.287258 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7" (UID: "3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:46:16.387927 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:16.387889 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:17.036661 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:17.036631 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" Jul 24 15:46:17.036841 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:17.036630 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz" event={"ID":"3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7","Type":"ContainerDied","Data":"1368489c8ea989f21ba1b72824c5b1b0c961f5cd53d1f969a46e4e2b1946b47b"} Jul 24 15:46:17.036841 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:17.036759 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1368489c8ea989f21ba1b72824c5b1b0c961f5cd53d1f969a46e4e2b1946b47b" Jul 24 15:46:25.721428 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:25.721391 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj"] Jul 24 15:46:25.721837 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:25.721622 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" podUID="8c3ff614-b21c-4800-8d5d-3eb3e69df761" containerName="maas-api" containerID="cri-o://63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b" gracePeriod=30 Jul 24 15:46:26.003445 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.003417 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:46:26.077005 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.076977 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3ff614-b21c-4800-8d5d-3eb3e69df761-maas-api-tls\") pod \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " Jul 24 15:46:26.077201 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.077118 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdc46\" (UniqueName: \"kubernetes.io/projected/8c3ff614-b21c-4800-8d5d-3eb3e69df761-kube-api-access-rdc46\") pod \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " Jul 24 15:46:26.077201 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.077143 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ff614-b21c-4800-8d5d-3eb3e69df761-openshift-service-ca\") pod \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\" (UID: \"8c3ff614-b21c-4800-8d5d-3eb3e69df761\") " Jul 24 15:46:26.077609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.077562 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3ff614-b21c-4800-8d5d-3eb3e69df761-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c3ff614-b21c-4800-8d5d-3eb3e69df761" (UID: "8c3ff614-b21c-4800-8d5d-3eb3e69df761"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:46:26.078443 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.078415 2557 generic.go:356] "Generic (PLEG): container finished" podID="8c3ff614-b21c-4800-8d5d-3eb3e69df761" containerID="63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b" exitCode=0 Jul 24 15:46:26.078573 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.078483 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" Jul 24 15:46:26.078573 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.078485 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" event={"ID":"8c3ff614-b21c-4800-8d5d-3eb3e69df761","Type":"ContainerDied","Data":"63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b"} Jul 24 15:46:26.078573 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.078552 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj" event={"ID":"8c3ff614-b21c-4800-8d5d-3eb3e69df761","Type":"ContainerDied","Data":"946315060e4d2f353bc4ea56b3377aa3c98d2e85845a8ee86882ee451a8f7f46"} Jul 24 15:46:26.078737 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.078577 2557 scope.go:117] "RemoveContainer" containerID="63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b" Jul 24 15:46:26.079559 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.079524 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3ff614-b21c-4800-8d5d-3eb3e69df761-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c3ff614-b21c-4800-8d5d-3eb3e69df761" (UID: "8c3ff614-b21c-4800-8d5d-3eb3e69df761"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:46:26.079722 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.079678 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3ff614-b21c-4800-8d5d-3eb3e69df761-kube-api-access-rdc46" (OuterVolumeSpecName: "kube-api-access-rdc46") pod "8c3ff614-b21c-4800-8d5d-3eb3e69df761" (UID: "8c3ff614-b21c-4800-8d5d-3eb3e69df761"). InnerVolumeSpecName "kube-api-access-rdc46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:46:26.096895 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.096871 2557 scope.go:117] "RemoveContainer" containerID="63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b" Jul 24 15:46:26.097213 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:46:26.097173 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b\": container with ID starting with 63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b not found: ID does not exist" containerID="63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b" Jul 24 15:46:26.097279 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.097219 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b"} err="failed to get container status \"63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b\": rpc error: code = NotFound desc = could not find container \"63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b\": container with ID starting with 63e229f54b40a4303ec2b9d2178e205dae0386d6b445ff0d79b7362b4924d69b not found: ID does not exist" Jul 24 15:46:26.178638 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.178609 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdc46\" (UniqueName: \"kubernetes.io/projected/8c3ff614-b21c-4800-8d5d-3eb3e69df761-kube-api-access-rdc46\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:26.178638 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.178634 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ff614-b21c-4800-8d5d-3eb3e69df761-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:26.178638 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.178646 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3ff614-b21c-4800-8d5d-3eb3e69df761-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:26.401028 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.400996 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj"] Jul 24 15:46:26.405231 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:26.405208 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d419ce-5c85997bb7-vd7nj"] Jul 24 15:46:27.587715 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:27.587681 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c3ff614-b21c-4800-8d5d-3eb3e69df761" path="/var/lib/kubelet/pods/8c3ff614-b21c-4800-8d5d-3eb3e69df761/volumes" Jul 24 15:46:28.668591 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:28.668555 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz"] Jul 24 15:46:28.672127 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:28.672089 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d419ce-tn9fz"] Jul 24 15:46:29.587243 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:29.587209 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7" path="/var/lib/kubelet/pods/3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7/volumes" Jul 24 15:46:35.575142 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:35.575106 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb"] Jul 24 15:46:35.575635 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:35.575392 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" containerID="cri-o://4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5" gracePeriod=30 Jul 24 15:46:36.324142 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:36.324105 2557 patch_prober.go:28] interesting pod/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:46:36.324325 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:36.324196 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:46:40.828545 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.828521 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:46:40.909500 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909473 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-certs\") pod \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " Jul 24 15:46:40.909686 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909526 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndtqr\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-kube-api-access-ndtqr\") pod \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " Jul 24 15:46:40.909686 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909553 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-credential-socket\") pod \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " Jul 24 15:46:40.909686 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909593 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-socket\") pod \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " Jul 24 15:46:40.909686 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909620 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-data\") pod \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " Jul 24 15:46:40.909686 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909646 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istiod-ca-cert\") pod \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " Jul 24 15:46:40.909974 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909689 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-podinfo\") pod \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " Jul 24 15:46:40.909974 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909717 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-token\") pod \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " Jul 24 15:46:40.909974 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909747 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-envoy\") pod \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\" (UID: \"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4\") " Jul 24 15:46:40.909974 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909792 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" (UID: "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:40.909974 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909914 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" (UID: "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:40.909974 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.909939 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" (UID: "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:40.910289 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.910055 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" (UID: "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:46:40.910289 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.910064 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:40.910289 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.910101 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:40.910289 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.910120 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:40.910850 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.910825 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" (UID: "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:40.911965 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.911943 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-kube-api-access-ndtqr" (OuterVolumeSpecName: "kube-api-access-ndtqr") pod "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" (UID: "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4"). InnerVolumeSpecName "kube-api-access-ndtqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:46:40.912051 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.911975 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" (UID: "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:40.912483 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.912467 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" (UID: "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:46:40.912535 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:40.912491 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" (UID: "b58ec4dc-a49b-4c33-a9e9-a846d418d9e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:46:41.010764 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.010728 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtqr\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-kube-api-access-ndtqr\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:41.010764 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.010759 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:41.010764 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.010768 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:41.010764 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.010776 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:41.011027 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.010784 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:41.011027 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.010794 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:41.151628 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.151592 2557 generic.go:356] "Generic (PLEG): container finished" podID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerID="4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5" exitCode=0 Jul 24 15:46:41.151798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.151646 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" Jul 24 15:46:41.151798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.151661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" event={"ID":"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4","Type":"ContainerDied","Data":"4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5"} Jul 24 15:46:41.151798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.151721 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb" event={"ID":"b58ec4dc-a49b-4c33-a9e9-a846d418d9e4","Type":"ContainerDied","Data":"1b349da1da8820e7bcaa904b5524e42c82531c987bfe6694f938f90af807d8fb"} Jul 24 15:46:41.151798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.151754 2557 scope.go:117] "RemoveContainer" containerID="4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5" Jul 24 15:46:41.161087 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.161070 2557 scope.go:117] "RemoveContainer" containerID="f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1" Jul 24 15:46:41.168751 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.168732 2557 scope.go:117] "RemoveContainer" containerID="4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5" Jul 24 15:46:41.168985 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:46:41.168967 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5\": container with ID starting with 4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5 not found: ID does not exist" containerID="4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5" Jul 24 15:46:41.169039 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.168992 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5"} err="failed to get container status \"4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5\": rpc error: code = NotFound desc = could not find container \"4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5\": container with ID starting with 4d8304a409407a3d9c1721d66521aabe31246fbb3a757a6f5eaa089c48a090a5 not found: ID does not exist" Jul 24 15:46:41.169039 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.169010 2557 scope.go:117] "RemoveContainer" containerID="f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1" Jul 24 15:46:41.169230 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:46:41.169211 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1\": container with ID starting with f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1 not found: ID does not exist" containerID="f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1" Jul 24 15:46:41.169278 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.169235 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1"} err="failed to get container status \"f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1\": rpc error: code = NotFound desc = could not find container \"f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1\": container with ID starting with f3a681c6d69b6afccea1c99414c6588d1833b0e4e26c89eff1dc85a9f444d5e1 not found: ID does not exist" Jul 24 15:46:41.175652 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.175628 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb"] Jul 24 15:46:41.179541 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.179517 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-d419ce-openshift-default-7f7f5448fb-z4zkb"] Jul 24 15:46:41.593549 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:41.593469 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" path="/var/lib/kubelet/pods/b58ec4dc-a49b-4c33-a9e9-a846d418d9e4/volumes" Jul 24 15:46:46.292160 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.292128 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk"] Jul 24 15:46:46.292748 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.292347 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" podUID="951c948c-9aba-4b39-9584-6d6b713a7859" containerName="maas-api" containerID="cri-o://7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781" gracePeriod=30 Jul 24 15:46:46.556944 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.556915 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:46:46.663869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.663832 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951c948c-9aba-4b39-9584-6d6b713a7859-openshift-service-ca\") pod \"951c948c-9aba-4b39-9584-6d6b713a7859\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " Jul 24 15:46:46.663869 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.663876 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951c948c-9aba-4b39-9584-6d6b713a7859-maas-api-tls\") pod \"951c948c-9aba-4b39-9584-6d6b713a7859\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " Jul 24 15:46:46.664177 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.663912 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n849\" (UniqueName: \"kubernetes.io/projected/951c948c-9aba-4b39-9584-6d6b713a7859-kube-api-access-4n849\") pod \"951c948c-9aba-4b39-9584-6d6b713a7859\" (UID: \"951c948c-9aba-4b39-9584-6d6b713a7859\") " Jul 24 15:46:46.664315 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.664274 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951c948c-9aba-4b39-9584-6d6b713a7859-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "951c948c-9aba-4b39-9584-6d6b713a7859" (UID: "951c948c-9aba-4b39-9584-6d6b713a7859"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:46:46.666329 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.666296 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951c948c-9aba-4b39-9584-6d6b713a7859-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "951c948c-9aba-4b39-9584-6d6b713a7859" (UID: "951c948c-9aba-4b39-9584-6d6b713a7859"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:46:46.666464 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.666442 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951c948c-9aba-4b39-9584-6d6b713a7859-kube-api-access-4n849" (OuterVolumeSpecName: "kube-api-access-4n849") pod "951c948c-9aba-4b39-9584-6d6b713a7859" (UID: "951c948c-9aba-4b39-9584-6d6b713a7859"). InnerVolumeSpecName "kube-api-access-4n849". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:46:46.764913 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.764879 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n849\" (UniqueName: \"kubernetes.io/projected/951c948c-9aba-4b39-9584-6d6b713a7859-kube-api-access-4n849\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:46.764913 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.764909 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951c948c-9aba-4b39-9584-6d6b713a7859-openshift-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:46.765118 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:46.764923 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951c948c-9aba-4b39-9584-6d6b713a7859-maas-api-tls\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:46:47.181374 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.181333 2557 generic.go:356] "Generic (PLEG): container finished" podID="951c948c-9aba-4b39-9584-6d6b713a7859" containerID="7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781" exitCode=0 Jul 24 15:46:47.181548 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.181380 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" Jul 24 15:46:47.181548 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.181413 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" event={"ID":"951c948c-9aba-4b39-9584-6d6b713a7859","Type":"ContainerDied","Data":"7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781"} Jul 24 15:46:47.181548 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.181460 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk" event={"ID":"951c948c-9aba-4b39-9584-6d6b713a7859","Type":"ContainerDied","Data":"79a4051ac8ec44c4dbba1c86819f3f0a5c15146ae5dbe126e0806ff05c8435f3"} Jul 24 15:46:47.181548 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.181483 2557 scope.go:117] "RemoveContainer" containerID="7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781" Jul 24 15:46:47.190298 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.190283 2557 scope.go:117] "RemoveContainer" containerID="7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781" Jul 24 15:46:47.190509 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:46:47.190492 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781\": container with ID starting with 7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781 not found: ID does not exist" containerID="7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781" Jul 24 15:46:47.190562 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.190517 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781"} err="failed to get container status \"7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781\": rpc error: code = NotFound desc = could not find container \"7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781\": container with ID starting with 7037e737bbaf676410c659a5881c14feebf28295f67ff7c2c967957c13245781 not found: ID does not exist" Jul 24 15:46:47.203361 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.203339 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk"] Jul 24 15:46:47.206780 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.206760 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6b942c-5f6dd59696-46mpk"] Jul 24 15:46:47.591667 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:47.591587 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="951c948c-9aba-4b39-9584-6d6b713a7859" path="/var/lib/kubelet/pods/951c948c-9aba-4b39-9584-6d6b713a7859/volumes" Jul 24 15:46:57.940172 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:57.940137 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m"] Jul 24 15:46:57.940761 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:46:57.940443 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" containerID="cri-o://6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1" gracePeriod=30 Jul 24 15:47:03.200585 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.200555 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:47:03.256160 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.256071 2557 generic.go:356] "Generic (PLEG): container finished" podID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerID="6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1" exitCode=0 Jul 24 15:47:03.256160 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.256126 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" Jul 24 15:47:03.256387 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.256156 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" event={"ID":"ecc9d46d-cd70-42d0-87d9-97b9e750375e","Type":"ContainerDied","Data":"6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1"} Jul 24 15:47:03.256387 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.256239 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m" event={"ID":"ecc9d46d-cd70-42d0-87d9-97b9e750375e","Type":"ContainerDied","Data":"5501a50f285bc2185f321f54b0241b83a9acd57825d1636139fae5eb51daaadf"} Jul 24 15:47:03.256387 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.256267 2557 scope.go:117] "RemoveContainer" containerID="6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1" Jul 24 15:47:03.264823 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.264801 2557 scope.go:117] "RemoveContainer" containerID="6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec" Jul 24 15:47:03.272907 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.272887 2557 scope.go:117] "RemoveContainer" containerID="6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1" Jul 24 15:47:03.273219 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:47:03.273198 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1\": container with ID starting with 6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1 not found: ID does not exist" containerID="6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1" Jul 24 15:47:03.273268 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.273232 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1"} err="failed to get container status \"6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1\": rpc error: code = NotFound desc = could not find container \"6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1\": container with ID starting with 6f3815101004592712d1866e5f0480b17f33ba0c0b024a6d45aa68f2bbdf9db1 not found: ID does not exist" Jul 24 15:47:03.273268 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.273255 2557 scope.go:117] "RemoveContainer" containerID="6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec" Jul 24 15:47:03.273510 ip-10-0-139-100 kubenswrapper[2557]: E0724 15:47:03.273492 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec\": container with ID starting with 6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec not found: ID does not exist" containerID="6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec" Jul 24 15:47:03.273574 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.273518 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec"} err="failed to get container status \"6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec\": rpc error: code = NotFound desc = could not find container \"6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec\": container with ID starting with 6a188bfd06076a725f4ddf7056614566f513225cb214813566611094c77b7bec not found: ID does not exist" Jul 24 15:47:03.311959 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.311919 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-token\") pod \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " Jul 24 15:47:03.312067 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.311981 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgms6\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-kube-api-access-bgms6\") pod \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " Jul 24 15:47:03.312067 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312012 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-envoy\") pod \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " Jul 24 15:47:03.312168 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312093 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istiod-ca-cert\") pod \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " Jul 24 15:47:03.312168 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312149 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-data\") pod \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " Jul 24 15:47:03.312306 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312173 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-podinfo\") pod \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " Jul 24 15:47:03.312306 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312233 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-socket\") pod \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " Jul 24 15:47:03.312306 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312260 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-credential-socket\") pod \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " Jul 24 15:47:03.312306 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312291 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-certs\") pod \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\" (UID: \"ecc9d46d-cd70-42d0-87d9-97b9e750375e\") " Jul 24 15:47:03.312565 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312515 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ecc9d46d-cd70-42d0-87d9-97b9e750375e" (UID: "ecc9d46d-cd70-42d0-87d9-97b9e750375e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:03.312720 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312587 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ecc9d46d-cd70-42d0-87d9-97b9e750375e" (UID: "ecc9d46d-cd70-42d0-87d9-97b9e750375e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:47:03.312720 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312702 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ecc9d46d-cd70-42d0-87d9-97b9e750375e" (UID: "ecc9d46d-cd70-42d0-87d9-97b9e750375e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:03.312829 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.312747 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ecc9d46d-cd70-42d0-87d9-97b9e750375e" (UID: "ecc9d46d-cd70-42d0-87d9-97b9e750375e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:03.313304 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.313275 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-data" (OuterVolumeSpecName: "istio-data") pod "ecc9d46d-cd70-42d0-87d9-97b9e750375e" (UID: "ecc9d46d-cd70-42d0-87d9-97b9e750375e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:03.314603 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.314581 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-token" (OuterVolumeSpecName: "istio-token") pod "ecc9d46d-cd70-42d0-87d9-97b9e750375e" (UID: "ecc9d46d-cd70-42d0-87d9-97b9e750375e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:47:03.314706 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.314579 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ecc9d46d-cd70-42d0-87d9-97b9e750375e" (UID: "ecc9d46d-cd70-42d0-87d9-97b9e750375e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:47:03.314860 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.314835 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ecc9d46d-cd70-42d0-87d9-97b9e750375e" (UID: "ecc9d46d-cd70-42d0-87d9-97b9e750375e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:03.314860 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.314845 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-kube-api-access-bgms6" (OuterVolumeSpecName: "kube-api-access-bgms6") pod "ecc9d46d-cd70-42d0-87d9-97b9e750375e" (UID: "ecc9d46d-cd70-42d0-87d9-97b9e750375e"). InnerVolumeSpecName "kube-api-access-bgms6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:47:03.413365 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.413329 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-data\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:47:03.413365 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.413358 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-podinfo\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:47:03.413365 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.413367 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:47:03.413621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.413377 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-credential-socket\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:47:03.413621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.413386 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-workload-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:47:03.413621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.413397 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-token\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:47:03.413621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.413405 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgms6\" (UniqueName: \"kubernetes.io/projected/ecc9d46d-cd70-42d0-87d9-97b9e750375e-kube-api-access-bgms6\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:47:03.413621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.413414 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istio-envoy\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:47:03.413621 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.413422 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9d46d-cd70-42d0-87d9-97b9e750375e-istiod-ca-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\"" Jul 24 15:47:03.579756 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.579723 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m"] Jul 24 15:47:03.587755 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:03.587728 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-6b942c-openshift-default-7b859d4f96-zht8m"] Jul 24 15:47:04.496363 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:04.496335 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d8bd4bd76-z5rrt_c229da9c-96e7-4655-85cc-2ea51565f23d/manager/0.log" Jul 24 15:47:04.744817 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:04.744788 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cx55j_a3e11020-e4db-4714-99e4-3519a38e2ac5/manager/0.log" Jul 24 15:47:05.174816 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:05.174788 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5584866b-9b7vf_fe20b512-4076-4eea-b66b-e2b4216df9c1/manager/0.log" Jul 24 15:47:05.189814 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:05.189788 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5584866b-9b7vf_fe20b512-4076-4eea-b66b-e2b4216df9c1/copy-manifests/0.log" Jul 24 15:47:05.588169 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:05.588088 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" path="/var/lib/kubelet/pods/ecc9d46d-cd70-42d0-87d9-97b9e750375e/volumes" Jul 24 15:47:05.673956 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:05.673932 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748435-h6rxq_608a814d-d262-475e-9fed-9ebdb29dccee/cleanup/0.log" Jul 24 15:47:05.788757 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:05.788731 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748450-4m5mp_349676d2-73cd-4a7c-935c-3be4bb5d9593/cleanup/0.log" Jul 24 15:47:05.903600 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:05.903579 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748465-hvdkb_21d7a321-59ed-4af3-9374-ca9e4147869e/cleanup/0.log" Jul 24 15:47:06.022818 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:06.022792 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kpqp4_8c989cb2-ce6e-4ea6-baf6-482f737eadb3/postgres/0.log" Jul 24 15:47:08.711521 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:08.711486 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd_f7482f19-8e2f-42c4-81cf-cc12517a6c18/istio-proxy/0.log" Jul 24 15:47:08.846896 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:08.846845 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6t92n_4829a10c-fab4-4195-8d5f-0cdbd8b90ccb/discovery/0.log" Jul 24 15:47:10.079248 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:10.079215 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fgk6c_a359727e-e692-46b0-8beb-ac46a99c4629/main/0.log" Jul 24 15:47:19.052892 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:19.052802 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ghk9t_bd1f81ff-a1c3-4714-bbec-423bdf6e06b9/global-pull-secret-syncer/0.log" Jul 24 15:47:19.187622 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:19.187591 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mgxft_25d523dd-9ec8-4ae7-84f7-22c587ab5fbd/konnectivity-agent/0.log" Jul 24 15:47:19.300827 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:19.300782 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-100.ec2.internal_a3ba7cef95532ae9bb2420cca3ac89de/haproxy/0.log" Jul 24 15:47:27.662355 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:27.662319 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-68cf457585-bqln9_55497eba-30a8-4b79-a109-bce0b5ca87c8/metrics-server/0.log" Jul 24 15:47:28.001798 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.001708 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cn668_d640e5ab-e9d6-43c6-8081-468e57639679/node-exporter/0.log" Jul 24 15:47:28.045231 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.045198 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cn668_d640e5ab-e9d6-43c6-8081-468e57639679/kube-rbac-proxy/0.log" Jul 24 15:47:28.074665 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.074632 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cn668_d640e5ab-e9d6-43c6-8081-468e57639679/init-textfile/0.log" Jul 24 15:47:28.769916 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.769886 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-8ntc9_ddaa3b2c-a46c-46d2-a805-b5f5b7a55707/prometheus-operator-admission-webhook/0.log" Jul 24 15:47:28.836441 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.836392 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-844c66b9bd-sltwr_ffc0eb56-9c0f-45a8-b210-3592e268338e/telemeter-client/0.log" Jul 24 15:47:28.904388 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904353 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl"] Jul 24 15:47:28.904765 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904747 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7" containerName="revoke-keys" Jul 24 15:47:28.904855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904768 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7" containerName="revoke-keys" Jul 24 15:47:28.904855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904784 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" Jul 24 15:47:28.904855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904792 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" Jul 24 15:47:28.904855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904803 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" Jul 24 15:47:28.904855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904812 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" Jul 24 15:47:28.904855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904830 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" Jul 24 15:47:28.904855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904839 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" Jul 24 15:47:28.904855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904849 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" Jul 24 15:47:28.904855 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904857 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904871 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951c948c-9aba-4b39-9584-6d6b713a7859" containerName="maas-api" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904880 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="951c948c-9aba-4b39-9584-6d6b713a7859" containerName="maas-api" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904892 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3ff614-b21c-4800-8d5d-3eb3e69df761" containerName="maas-api" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904900 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3ff614-b21c-4800-8d5d-3eb3e69df761" containerName="maas-api" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904984 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.904995 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc9d46d-cd70-42d0-87d9-97b9e750375e" containerName="istio-proxy" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.905005 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.905014 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="951c948c-9aba-4b39-9584-6d6b713a7859" containerName="maas-api" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.905024 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3ff614-b21c-4800-8d5d-3eb3e69df761" containerName="maas-api" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.905033 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58ec4dc-a49b-4c33-a9e9-a846d418d9e4" containerName="istio-proxy" Jul 24 15:47:28.905342 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.905041 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a69c1d2-ce6d-4a6c-b5a6-56315b7081a7" containerName="revoke-keys" Jul 24 15:47:28.910218 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.910175 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:28.914724 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.914706 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fd7vt\"/\"kube-root-ca.crt\"" Jul 24 15:47:28.915305 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.915288 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fd7vt\"/\"openshift-service-ca.crt\"" Jul 24 15:47:28.915955 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.915937 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fd7vt\"/\"default-dockercfg-sprr5\"" Jul 24 15:47:28.929080 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.929053 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl"] Jul 24 15:47:28.936121 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.936097 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-844c66b9bd-sltwr_ffc0eb56-9c0f-45a8-b210-3592e268338e/reload/0.log" Jul 24 15:47:28.995113 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:28.995089 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-844c66b9bd-sltwr_ffc0eb56-9c0f-45a8-b210-3592e268338e/kube-rbac-proxy/0.log" Jul 24 15:47:29.037299 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.037202 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjjh\" (UniqueName: \"kubernetes.io/projected/5339cb05-78f9-4865-b895-bb58a25527ed-kube-api-access-pmjjh\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.037299 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.037240 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-sys\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.037299 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.037264 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-proc\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.037299 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.037297 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-podres\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.037567 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.037329 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-lib-modules\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.048800 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.048775 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49757f9c-wk6pd_f1ccc53f-2d30-4cda-aab9-9da65207fd8b/thanos-query/0.log" Jul 24 15:47:29.099847 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.099820 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49757f9c-wk6pd_f1ccc53f-2d30-4cda-aab9-9da65207fd8b/kube-rbac-proxy-web/0.log" Jul 24 15:47:29.138776 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.138731 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-lib-modules\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.138960 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.138835 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjjh\" (UniqueName: \"kubernetes.io/projected/5339cb05-78f9-4865-b895-bb58a25527ed-kube-api-access-pmjjh\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.138960 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.138864 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-sys\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.138960 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.138896 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-proc\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.138960 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.138907 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-lib-modules\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.138960 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.138925 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-podres\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.139170 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.138986 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-sys\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.139170 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.139008 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-podres\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.139170 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.138988 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5339cb05-78f9-4865-b895-bb58a25527ed-proc\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.151338 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.151258 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjjh\" (UniqueName: \"kubernetes.io/projected/5339cb05-78f9-4865-b895-bb58a25527ed-kube-api-access-pmjjh\") pod \"perf-node-gather-daemonset-b27pl\" (UID: \"5339cb05-78f9-4865-b895-bb58a25527ed\") " pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.152099 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.152051 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49757f9c-wk6pd_f1ccc53f-2d30-4cda-aab9-9da65207fd8b/kube-rbac-proxy/0.log" Jul 24 15:47:29.194725 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.194693 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49757f9c-wk6pd_f1ccc53f-2d30-4cda-aab9-9da65207fd8b/prom-label-proxy/0.log" Jul 24 15:47:29.219926 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.219893 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:29.224761 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.224740 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49757f9c-wk6pd_f1ccc53f-2d30-4cda-aab9-9da65207fd8b/kube-rbac-proxy-rules/0.log" Jul 24 15:47:29.261887 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.261855 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49757f9c-wk6pd_f1ccc53f-2d30-4cda-aab9-9da65207fd8b/kube-rbac-proxy-metrics/0.log" Jul 24 15:47:29.361365 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.361336 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl"] Jul 24 15:47:29.365646 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:47:29.365623 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5339cb05_78f9_4865_b895_bb58a25527ed.slice/crio-f3e39e496eb76745061ae4ca78fae82ceec6e2506f442cdf1899a33a79823670 WatchSource:0}: Error finding container f3e39e496eb76745061ae4ca78fae82ceec6e2506f442cdf1899a33a79823670: Status 404 returned error can't find the container with id f3e39e496eb76745061ae4ca78fae82ceec6e2506f442cdf1899a33a79823670 Jul 24 15:47:29.377562 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:29.377534 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" event={"ID":"5339cb05-78f9-4865-b895-bb58a25527ed","Type":"ContainerStarted","Data":"f3e39e496eb76745061ae4ca78fae82ceec6e2506f442cdf1899a33a79823670"} Jul 24 15:47:30.383609 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:30.383575 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" event={"ID":"5339cb05-78f9-4865-b895-bb58a25527ed","Type":"ContainerStarted","Data":"6da0cfaff68ce1dea51636c7c297a3c8d70e1eb17728e99d64f88e1b0bd7229c"} Jul 24 15:47:30.384079 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:30.383697 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:30.408371 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:30.408320 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" podStartSLOduration=2.408302517 podStartE2EDuration="2.408302517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:30.406648326 +0000 UTC m=+3069.354141913" watchObservedRunningTime="2026-07-24 15:47:30.408302517 +0000 UTC m=+3069.355796144" Jul 24 15:47:33.549570 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:33.549538 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vlbn_8a312eb8-60f9-4e4c-8409-be38cb7fbbbb/dns/0.log" Jul 24 15:47:33.580717 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:33.580694 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vlbn_8a312eb8-60f9-4e4c-8409-be38cb7fbbbb/kube-rbac-proxy/0.log" Jul 24 15:47:33.701867 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:33.701837 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t5h8r_a5f95e57-a279-4df4-a232-1d264a35c3fb/dns-node-resolver/0.log" Jul 24 15:47:34.449016 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:34.448985 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s2rlr_668e6b2b-8552-4dc8-a40d-3d7961a9b066/node-ca/0.log" Jul 24 15:47:35.553931 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:35.553901 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd_f7482f19-8e2f-42c4-81cf-cc12517a6c18/istio-proxy/0.log" Jul 24 15:47:35.705674 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:35.705638 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6t92n_4829a10c-fab4-4195-8d5f-0cdbd8b90ccb/discovery/0.log" Jul 24 15:47:36.398301 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:36.398275 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fd7vt/perf-node-gather-daemonset-b27pl" Jul 24 15:47:36.816202 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:36.816117 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d2f7n_17960c0a-1d61-4c0f-aa87-2aed37d99494/serve-healthcheck-canary/0.log" Jul 24 15:47:37.487719 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:37.487684 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hx2m4_2abba8f0-87ce-4adf-b88b-29915e84a3b8/kube-rbac-proxy/0.log" Jul 24 15:47:37.527284 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:37.527257 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hx2m4_2abba8f0-87ce-4adf-b88b-29915e84a3b8/exporter/0.log" Jul 24 15:47:37.565401 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:37.565375 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hx2m4_2abba8f0-87ce-4adf-b88b-29915e84a3b8/extractor/0.log" Jul 24 15:47:39.784670 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:39.784642 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd_f7482f19-8e2f-42c4-81cf-cc12517a6c18/istio-proxy/0.log" Jul 24 15:47:39.923260 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:39.923232 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6t92n_4829a10c-fab4-4195-8d5f-0cdbd8b90ccb/discovery/0.log" Jul 24 15:47:40.798774 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:40.798743 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d8bd4bd76-z5rrt_c229da9c-96e7-4655-85cc-2ea51565f23d/manager/0.log" Jul 24 15:47:40.924304 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:40.924224 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cx55j_a3e11020-e4db-4714-99e4-3519a38e2ac5/manager/0.log" Jul 24 15:47:41.154673 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:41.154644 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5584866b-9b7vf_fe20b512-4076-4eea-b66b-e2b4216df9c1/manager/0.log" Jul 24 15:47:41.208947 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:41.208870 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5584866b-9b7vf_fe20b512-4076-4eea-b66b-e2b4216df9c1/copy-manifests/0.log" Jul 24 15:47:41.496097 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:41.496023 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rf9xd_f7482f19-8e2f-42c4-81cf-cc12517a6c18/istio-proxy/0.log" Jul 24 15:47:41.636973 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:41.636939 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6t92n_4829a10c-fab4-4195-8d5f-0cdbd8b90ccb/discovery/0.log" Jul 24 15:47:42.482637 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:42.482603 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54d8b9c4f-lnkkn_b4b953b5-1229-4969-b243-226a97234d6a/manager/0.log" Jul 24 15:47:44.520217 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.520173 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-48zpt"] Jul 24 15:47:44.528751 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.528729 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-48zpt" Jul 24 15:47:44.532488 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.532461 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:47:44.533064 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.533043 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-mjwv9\"" Jul 24 15:47:44.533361 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.533346 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:47:44.534339 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.534320 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-48zpt"] Jul 24 15:47:44.680523 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.680485 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96tx\" (UniqueName: \"kubernetes.io/projected/87f7a87d-95b8-4f49-b8ce-026d718e38db-kube-api-access-q96tx\") pod \"kuadrant-operator-catalog-48zpt\" (UID: \"87f7a87d-95b8-4f49-b8ce-026d718e38db\") " pod="kuadrant-system/kuadrant-operator-catalog-48zpt" Jul 24 15:47:44.782101 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.782017 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96tx\" (UniqueName: \"kubernetes.io/projected/87f7a87d-95b8-4f49-b8ce-026d718e38db-kube-api-access-q96tx\") pod \"kuadrant-operator-catalog-48zpt\" (UID: \"87f7a87d-95b8-4f49-b8ce-026d718e38db\") " pod="kuadrant-system/kuadrant-operator-catalog-48zpt" Jul 24 15:47:44.794531 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.794500 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96tx\" (UniqueName: \"kubernetes.io/projected/87f7a87d-95b8-4f49-b8ce-026d718e38db-kube-api-access-q96tx\") pod \"kuadrant-operator-catalog-48zpt\" (UID: \"87f7a87d-95b8-4f49-b8ce-026d718e38db\") " pod="kuadrant-system/kuadrant-operator-catalog-48zpt" Jul 24 15:47:44.838482 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.838464 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-48zpt" Jul 24 15:47:44.985735 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.985700 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-48zpt"] Jul 24 15:47:44.988867 ip-10-0-139-100 kubenswrapper[2557]: W0724 15:47:44.988837 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f7a87d_95b8_4f49_b8ce_026d718e38db.slice/crio-045a19e566c546cdf240225315dd1cdff4d476ac5328678409e21199af1215e8 WatchSource:0}: Error finding container 045a19e566c546cdf240225315dd1cdff4d476ac5328678409e21199af1215e8: Status 404 returned error can't find the container with id 045a19e566c546cdf240225315dd1cdff4d476ac5328678409e21199af1215e8 Jul 24 15:47:44.990750 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:44.990731 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:47:45.456803 ip-10-0-139-100 kubenswrapper[2557]: I0724 15:47:45.456768 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-48zpt" event={"ID":"87f7a87d-95b8-4f49-b8ce-026d718e38db","Type":"ContainerStarted","Data":"045a19e566c546cdf240225315dd1cdff4d476ac5328678409e21199af1215e8"}