Jul 20 22:51:41.577764 ip-10-0-129-209 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 20 22:51:41.577776 ip-10-0-129-209 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 20 22:51:41.577783 ip-10-0-129-209 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 20 22:51:41.578053 ip-10-0-129-209 systemd[1]: Failed to start Kubernetes Kubelet. Jul 20 22:51:51.819082 ip-10-0-129-209 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 20 22:51:51.819107 ip-10-0-129-209 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 713255a6c68f496aa63d64306285c28f -- Jul 20 22:53:44.589744 ip-10-0-129-209 systemd[1]: Starting Kubernetes Kubelet... Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.918147 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925262 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925278 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925284 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925289 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925293 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925297 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925302 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:44.984748 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925306 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925311 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925315 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925319 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925323 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925327 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925335 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925341 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925346 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925351 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925355 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925358 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925362 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925366 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925370 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925374 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925377 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925381 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925385 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:44.985592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925389 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925393 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925396 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925400 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925412 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925416 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925419 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925423 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925427 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925430 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925434 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925438 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925441 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925445 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925448 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925452 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925470 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925484 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925490 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925494 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:44.986264 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925498 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925503 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925509 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925514 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925517 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925521 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925525 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925529 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925533 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925536 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925540 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925544 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925548 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925552 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925556 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925560 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925564 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925575 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925578 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925582 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:44.986945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925586 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925589 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925593 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925597 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925601 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925605 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925609 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925613 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925617 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925621 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925625 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925629 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925633 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925637 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925641 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925645 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925650 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925654 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925658 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925661 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:44.987837 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925665 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925669 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925673 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925677 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925681 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925685 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925688 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925692 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925697 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925702 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925716 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925721 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.925726 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926816 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926825 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926830 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926836 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926842 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926847 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:44.988919 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926852 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926856 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926859 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926863 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926867 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926871 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926875 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926881 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926885 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926890 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926895 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926899 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926906 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926911 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926915 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926918 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926923 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926926 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926930 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:44.989767 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926934 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926938 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926941 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926946 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926949 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926954 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926958 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926962 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926966 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926970 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926974 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926978 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926982 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926986 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926989 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926993 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.926997 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927001 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927005 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927009 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:44.990393 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927013 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927017 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927021 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927025 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927029 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927033 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927037 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927041 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927046 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927050 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927053 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927058 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927061 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927065 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927069 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927072 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927076 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927081 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927090 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927093 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:44.991030 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927097 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927101 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927105 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927112 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927116 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927119 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927123 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927127 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927131 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927135 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927139 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927143 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927146 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927150 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927154 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927158 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927162 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927166 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927170 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927174 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:44.991663 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927177 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927181 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927184 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927188 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927191 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927195 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927198 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927202 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927206 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927210 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927213 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927218 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927222 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927226 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927800 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927808 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927812 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927817 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927821 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927824 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:44.992301 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927828 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927832 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927836 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927840 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927844 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927848 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927852 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927856 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927859 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927863 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927867 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927871 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927875 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927880 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927883 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927887 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927892 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927895 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927899 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927904 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:44.995322 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927907 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927911 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927918 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927923 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927928 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927932 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927937 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927943 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927947 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927952 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927955 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927959 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927964 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927967 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927971 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927975 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927979 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927983 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927988 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:44.995947 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927992 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927996 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.927999 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928003 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928007 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928010 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928014 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928021 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928025 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928029 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928033 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928037 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928041 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928044 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928050 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928053 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928057 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928061 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928065 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:44.996783 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928068 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:44.997197 ip-10-0-129-209 systemd[1]: Started Kubernetes Kubelet. Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928073 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928077 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928081 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928085 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928089 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928092 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928095 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928099 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928105 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928108 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928112 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928116 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928120 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928123 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928127 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928131 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928135 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928138 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928142 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:44.997977 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928146 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928149 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928153 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928157 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928160 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928164 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928168 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928172 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928175 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928179 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928183 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928187 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928191 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928197 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.928200 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928290 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928312 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928320 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928326 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928332 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928337 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 22:53:44.998882 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928345 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928351 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928356 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928361 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928366 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928370 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928375 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928379 2574 flags.go:64] FLAG: --cgroup-root="" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928384 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928389 2574 flags.go:64] FLAG: --client-ca-file="" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928393 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928397 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928403 2574 flags.go:64] FLAG: --cluster-domain="" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928407 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928411 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928416 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928422 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928428 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928433 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928437 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928442 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928446 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928450 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928455 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 22:53:44.999718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928475 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928480 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928486 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928491 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928495 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928499 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928504 2574 flags.go:64] FLAG: --enable-server="true" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928508 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928516 2574 flags.go:64] FLAG: --event-burst="100" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928521 2574 flags.go:64] FLAG: --event-qps="50" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928527 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928531 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928535 2574 flags.go:64] FLAG: --eviction-hard="" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928541 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928545 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928550 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928554 2574 flags.go:64] FLAG: --eviction-soft="" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928559 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928564 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928568 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928572 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928577 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928581 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928585 2574 flags.go:64] FLAG: --feature-gates="" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928591 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 22:53:45.000888 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928595 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928600 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928605 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928609 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928614 2574 flags.go:64] FLAG: --help="false" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928618 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928623 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928629 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928634 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928638 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928644 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928647 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928651 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928656 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928659 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928665 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928670 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928674 2574 flags.go:64] FLAG: --kube-reserved="" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928679 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928683 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928687 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928692 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928696 2574 flags.go:64] FLAG: --lock-file="" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928700 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 22:53:45.002107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928704 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928708 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928717 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928721 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928726 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928731 2574 flags.go:64] FLAG: --logging-format="text" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928734 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928739 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928744 2574 flags.go:64] FLAG: --manifest-url="" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928747 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928753 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928758 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928764 2574 flags.go:64] FLAG: --max-pods="110" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928769 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928773 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928778 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928783 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928788 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928793 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928797 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928815 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928819 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928824 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 22:53:45.006935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928830 2574 flags.go:64] FLAG: --pod-cidr="" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928834 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928842 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928846 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928850 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928855 2574 flags.go:64] FLAG: --port="10250" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928859 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928863 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d724ce6f341994e" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928868 2574 flags.go:64] FLAG: --qos-reserved="" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928873 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928878 2574 flags.go:64] FLAG: --register-node="true" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928882 2574 flags.go:64] FLAG: --register-with-taints="" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928887 2574 flags.go:64] FLAG: --registry-burst="10" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928891 2574 flags.go:64] FLAG: --registry-qps="5" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928895 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928899 2574 flags.go:64] FLAG: --reserved-memory="" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928905 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928909 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928914 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928918 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928922 2574 flags.go:64] FLAG: --runonce="false" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928926 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928930 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928935 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928940 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 22:53:45.007782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928944 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928950 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928954 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928959 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928963 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928967 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928972 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928976 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928982 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928987 2574 flags.go:64] FLAG: --system-cgroups="" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928991 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.928998 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929002 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929007 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929012 2574 flags.go:64] FLAG: --tls-min-version="" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929017 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929021 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929026 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929030 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929035 2574 flags.go:64] FLAG: --v="2" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929041 2574 flags.go:64] FLAG: --version="false" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929047 2574 flags.go:64] FLAG: --vmodule="" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929053 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.929058 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929247 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:45.089320 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929252 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929257 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929261 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929265 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929269 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929273 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929280 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929284 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929288 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929296 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929300 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929304 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929307 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929311 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929315 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929320 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929323 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929327 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929331 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929334 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:45.090167 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929338 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929342 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929345 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929349 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929353 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929357 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929360 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929364 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929368 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929372 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929376 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929380 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929383 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929387 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929390 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929394 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929397 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929401 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929407 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929411 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:45.091052 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929414 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929418 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929424 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929429 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929433 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929437 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929441 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929447 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929451 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929472 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929478 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929482 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929486 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929491 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929495 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929499 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929505 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929509 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:45.091811 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929513 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929517 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929520 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929524 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929528 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929532 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929535 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929539 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929543 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929547 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929551 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929555 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929561 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929565 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929569 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929573 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929576 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929581 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929584 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929588 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:45.092402 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929592 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929597 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929601 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929604 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929608 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929612 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929616 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929620 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929624 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929628 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929631 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929635 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929639 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929642 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929646 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929650 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929653 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929657 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929661 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:45.093049 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.929664 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.930148 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.939845 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.939864 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.940618 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.940627 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940693 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940699 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940702 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940705 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940709 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940712 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940714 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940717 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940719 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:45.093613 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940722 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940724 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940727 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940730 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940733 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940736 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940738 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940741 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940744 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940746 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940749 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940753 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940758 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940761 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940765 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940768 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940770 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940773 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940776 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:45.094070 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940784 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940787 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940790 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940792 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940797 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940807 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940812 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940816 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940819 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940822 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940825 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940828 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940830 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940833 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940836 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940839 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940841 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940848 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940851 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:45.094639 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940853 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940856 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940859 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940863 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940866 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940869 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940872 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940875 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940878 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940881 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940884 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940887 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940889 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940892 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940896 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940898 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940901 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940904 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940906 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940909 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:45.095189 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940911 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940914 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940917 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940919 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940922 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940924 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940927 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940930 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940933 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940935 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940940 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940942 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940946 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940949 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940952 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940955 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940957 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940960 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940963 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940965 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:45.095735 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940969 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940971 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940974 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940977 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940979 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940983 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940985 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940988 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940991 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940993 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940996 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.940999 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.941005 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941136 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941141 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:45.096298 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941144 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941147 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941150 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941152 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941154 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941157 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941160 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941162 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941166 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941169 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941172 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941175 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941178 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941180 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941183 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941185 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941191 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941195 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941198 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941201 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:45.096726 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941204 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941207 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941210 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941213 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941215 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941218 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941221 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941224 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941227 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941229 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941232 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941235 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941237 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941240 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941243 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941245 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941248 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941250 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941253 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:45.097328 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941255 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941259 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941262 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941264 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941267 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941270 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941273 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941275 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941278 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941282 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941285 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941289 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941292 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941295 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941298 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941300 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941303 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941306 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941308 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:45.097848 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941311 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941313 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941316 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941318 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941321 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941324 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941326 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941328 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941331 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941333 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941336 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941339 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941341 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941343 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941347 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941350 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941352 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941355 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941358 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941360 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:45.098386 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941363 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941365 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941369 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941371 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941374 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941377 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941380 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941382 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941384 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941387 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941389 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941392 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941395 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941399 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941401 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941404 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941407 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941409 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:45.098945 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:44.941412 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.941417 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.941569 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.943568 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.944674 2574 server.go:1019] "Starting client certificate rotation" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.944775 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.944862 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.946642 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.950916 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.960135 2574 log.go:25] "Validated CRI v1 runtime API" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.969052 2574 log.go:25] "Validated CRI v1 image API" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.970901 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 22:53:45.099645 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.972914 2574 fs.go:135] Filesystem UUIDs: map[1466bdf2-5fc4-46bf-b55b-04323ddc7e12:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fab808f7-e9f0-48f6-b1fe-4c0703adcba4:/dev/nvme0n1p4] Jul 20 22:53:45.100056 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.972942 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 22:53:45.100174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.978148 2574 manager.go:216] Machine: {Timestamp:2026-07-20 22:53:44.977156047 +0000 UTC m=+0.295508324 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098136 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec231597a2a02ab14e0a7b4350f7a750 SystemUUID:ec231597-a2a0-2ab1-4e0a-7b4350f7a750 BootID:713255a6-c68f-496a-a63d-64306285c28f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2d:f9:fb:f4:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2d:f9:fb:f4:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:c9:66:8d:05:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 22:53:45.100174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.978269 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 22:53:45.100174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.978350 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 22:53:45.100174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.979196 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 22:53:45.100174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.979227 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-209.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 22:53:45.100174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.980283 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 22:53:45.100174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.980294 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 22:53:45.100174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.980302 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.980820 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.982020 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.984024 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.985770 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.985888 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.986376 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.986400 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.986425 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.986435 2574 kubelet.go:398] "Adding apiserver pod source" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.986673 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.987258 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.987276 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.989269 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.991122 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.991149 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992141 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992154 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992161 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992166 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992175 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992180 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992189 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992195 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992206 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992211 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992220 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992226 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992953 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.992963 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.996014 2574 server.go:1268] "Started kubelet" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.996566 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.997227 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.997285 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:44.998923 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.001149 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.014237 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.014429 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.014440 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.014853 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015091 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qn8vl" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015145 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015148 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015193 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 22:53:45.101354 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015277 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015286 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015325 2574 factory.go:55] Registering systemd factory Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015339 2574 factory.go:223] Registration of the systemd container factory successfully Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015595 2574 factory.go:153] Registering CRI-O factory Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015608 2574 factory.go:223] Registration of the crio container factory successfully Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015676 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015698 2574 factory.go:103] Registering Raw factory Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.015714 2574 manager.go:1196] Started watching for new ooms in manager Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.016143 2574 manager.go:349] Starting recovery of all containers Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.016826 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.017263 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qn8vl" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.017371 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-209.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.017402 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.016860 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-209.ec2.internal.18c4215a844d5471 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-209.ec2.internal,UID:ip-10-0-129-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-209.ec2.internal,},FirstTimestamp:2026-07-20 22:53:44.995992689 +0000 UTC m=+0.314344966,LastTimestamp:2026-07-20 22:53:44.995992689 +0000 UTC m=+0.314344966,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-209.ec2.internal,}" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.017771 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.017867 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.025633 2574 manager.go:354] Recovery completed Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.030357 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.032435 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.032485 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.032498 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.033049 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.033063 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.033051 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.033082 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.033867 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:45.103537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.033892 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.033905 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.034334 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-209.ec2.internal.18c4215a8679a59a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-209.ec2.internal,UID:ip-10-0-129-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-209.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-209.ec2.internal,},FirstTimestamp:2026-07-20 22:53:45.032451482 +0000 UTC m=+0.350803769,LastTimestamp:2026-07-20 22:53:45.032451482 +0000 UTC m=+0.350803769,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-209.ec2.internal,}" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.035342 2574 policy_none.go:49] "None policy: Start" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.035357 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.035367 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.037340 2574 policy_none.go:47] "Start" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.100210 2574 manager.go:337] "Starting Device Plugin manager" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.100412 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.100425 2574 server.go:87] "Starting device plugin registration server" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.100738 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.100748 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.100946 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.101001 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.101007 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.103362 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.103412 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.201675 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.202690 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.202723 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.202732 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.202758 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.210827 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.210848 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-209.ec2.internal\": node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.216069 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.221214 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.221257 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.221285 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.221322 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.223398 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:45.239393 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.224961 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.321531 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal"] Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.321615 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.323119 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.323150 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.323170 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.325237 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.325590 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.325690 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.325714 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.326389 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.326420 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.326429 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.326389 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.326496 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.326507 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.329561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.329791 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.331718 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.331749 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.331764 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.352116 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-209.ec2.internal\" not found" node="ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.366105 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.357214 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-209.ec2.internal\" not found" node="ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.444095 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.426446 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.529840 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.516739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd794cd41a0501a11ba5f67d42185203-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal\" (UID: \"bd794cd41a0501a11ba5f67d42185203\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.529840 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.516767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd794cd41a0501a11ba5f67d42185203-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal\" (UID: \"bd794cd41a0501a11ba5f67d42185203\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.529840 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.516791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f2e386cf084133e9c2b8d9bc35bd306-config\") pod \"kube-apiserver-proxy-ip-10-0-129-209.ec2.internal\" (UID: \"4f2e386cf084133e9c2b8d9bc35bd306\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.529840 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.526793 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.676627 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.617040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f2e386cf084133e9c2b8d9bc35bd306-config\") pod \"kube-apiserver-proxy-ip-10-0-129-209.ec2.internal\" (UID: \"4f2e386cf084133e9c2b8d9bc35bd306\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.676627 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.617069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd794cd41a0501a11ba5f67d42185203-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal\" (UID: \"bd794cd41a0501a11ba5f67d42185203\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.676627 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.617085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd794cd41a0501a11ba5f67d42185203-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal\" (UID: \"bd794cd41a0501a11ba5f67d42185203\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.676627 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.617121 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd794cd41a0501a11ba5f67d42185203-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal\" (UID: \"bd794cd41a0501a11ba5f67d42185203\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.676627 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.617126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f2e386cf084133e9c2b8d9bc35bd306-config\") pod \"kube-apiserver-proxy-ip-10-0-129-209.ec2.internal\" (UID: \"4f2e386cf084133e9c2b8d9bc35bd306\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.676627 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.617189 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd794cd41a0501a11ba5f67d42185203-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal\" (UID: \"bd794cd41a0501a11ba5f67d42185203\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.676627 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.627133 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.676627 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.656282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.676627 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.659774 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.748813 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.727579 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.851810 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:45.827986 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-209.ec2.internal\" not found" Jul 20 22:53:45.949917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.891533 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:45.949917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.915285 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.949917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.926467 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 22:53:45.949917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.927682 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal" Jul 20 22:53:45.949917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.936068 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 22:53:45.949917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.943992 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 22:53:45.949917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.944128 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:45.949917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.944093 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 22:53:45.949917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.944093 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.987267 2574 apiserver.go:52] "Watching apiserver" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.996325 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.996953 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-kxxtg","kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal","openshift-dns/node-resolver-jx6hc","openshift-image-registry/node-ca-cmrt4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal","openshift-multus/network-metrics-daemon-wpvj7","openshift-network-diagnostics/network-check-target-ff7jr","openshift-network-operator/iptables-alerter-xmtn4","kube-system/global-pull-secret-syncer-574vd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf","openshift-cluster-node-tuning-operator/tuned-7s78t","openshift-multus/multus-additional-cni-plugins-n8mn7","openshift-multus/multus-vpkxf","openshift-ovn-kubernetes/ovnkube-node-sjc24"] Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:45.999424 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.002136 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.002141 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.002158 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.002196 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z2ppv\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.004096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.004551 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nnr6s\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.004573 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.004620 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.006330 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.006391 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n9fvb\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.006495 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.006945 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.006950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.007009 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.009776 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.009837 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.012026 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.014674 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.014701 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.015406 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.015407 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 22:53:46.035128 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.015429 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tcvf4\"" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019264 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e84fa919-76a7-476e-9657-c195f0217bbc-hosts-file\") pod \"node-resolver-jx6hc\" (UID: \"e84fa919-76a7-476e-9657-c195f0217bbc\") " pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019301 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nv77\" (UniqueName: \"kubernetes.io/projected/7f29150c-6bc0-49cc-af95-18545831cf49-kube-api-access-7nv77\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5be5526a-0c29-42b3-91fb-6c6863746390-kubelet-config\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mftf\" (UniqueName: \"kubernetes.io/projected/e84fa919-76a7-476e-9657-c195f0217bbc-kube-api-access-2mftf\") pod \"node-resolver-jx6hc\" (UID: \"e84fa919-76a7-476e-9657-c195f0217bbc\") " pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b-host-slash\") pod \"iptables-alerter-xmtn4\" (UID: \"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b\") " pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36201deb-a807-451a-9a83-7446f325f7bc-agent-certs\") pod \"konnectivity-agent-kxxtg\" (UID: \"36201deb-a807-451a-9a83-7446f325f7bc\") " pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84fa919-76a7-476e-9657-c195f0217bbc-tmp-dir\") pod \"node-resolver-jx6hc\" (UID: \"e84fa919-76a7-476e-9657-c195f0217bbc\") " pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019447 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b-iptables-alerter-script\") pod \"iptables-alerter-xmtn4\" (UID: \"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b\") " pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36201deb-a807-451a-9a83-7446f325f7bc-konnectivity-ca\") pod \"konnectivity-agent-kxxtg\" (UID: \"36201deb-a807-451a-9a83-7446f325f7bc\") " pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0474b40b-5711-4a08-a644-f9caf4afed37-host\") pod \"node-ca-cmrt4\" (UID: \"0474b40b-5711-4a08-a644-f9caf4afed37\") " pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0474b40b-5711-4a08-a644-f9caf4afed37-serviceca\") pod \"node-ca-cmrt4\" (UID: \"0474b40b-5711-4a08-a644-f9caf4afed37\") " pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.038320 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019550 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5be5526a-0c29-42b3-91fb-6c6863746390-dbus\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6bc\" (UniqueName: \"kubernetes.io/projected/2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b-kube-api-access-9x6bc\") pod \"iptables-alerter-xmtn4\" (UID: \"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b\") " pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.019624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xswgq\" (UniqueName: \"kubernetes.io/projected/0474b40b-5711-4a08-a644-f9caf4afed37-kube-api-access-xswgq\") pod \"node-ca-cmrt4\" (UID: \"0474b40b-5711-4a08-a644-f9caf4afed37\") " pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.022839 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 22:48:44 +0000 UTC" deadline="2028-01-13 09:21:50.971479192 +0000 UTC" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.022869 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12994h28m4.948612521s" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.023419 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.023504 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.025739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.026575 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.027957 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.027961 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.027994 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.028034 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ft5lv\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.028359 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.030517 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8564l\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.030547 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.030589 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.030856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.033320 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.033327 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.033339 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.033528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.033550 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thj8l\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.033562 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 22:53:46.038900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.034085 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 22:53:46.041401 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.036278 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ft7p5\"" Jul 20 22:53:46.041401 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.038610 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 22:53:46.044415 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.044073 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wnp49" Jul 20 22:53:46.111668 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.046312 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.111668 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.049254 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 22:53:46.111668 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.049254 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 22:53:46.111668 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.049357 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 22:53:46.111668 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.049469 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pkmkh\"" Jul 20 22:53:46.111668 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.049474 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 22:53:46.111668 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.049473 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 22:53:46.111668 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.050581 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 22:53:46.111668 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.051034 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wnp49" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.116155 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.119870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-os-release\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.119892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-socket-dir-parent\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.119908 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.119922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-socket-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.119941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84fa919-76a7-476e-9657-c195f0217bbc-tmp-dir\") pod \"node-resolver-jx6hc\" (UID: \"e84fa919-76a7-476e-9657-c195f0217bbc\") " pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.119979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36201deb-a807-451a-9a83-7446f325f7bc-konnectivity-ca\") pod \"konnectivity-agent-kxxtg\" (UID: \"36201deb-a807-451a-9a83-7446f325f7bc\") " pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0474b40b-5711-4a08-a644-f9caf4afed37-host\") pod \"node-ca-cmrt4\" (UID: \"0474b40b-5711-4a08-a644-f9caf4afed37\") " pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bad4f19-6640-4c47-8340-af3bbaf3b736-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-conf-dir\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0474b40b-5711-4a08-a644-f9caf4afed37-host\") pod \"node-ca-cmrt4\" (UID: \"0474b40b-5711-4a08-a644-f9caf4afed37\") " pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-etc-kubernetes\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-kubelet\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-run-systemd\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.180203 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6bc\" (UniqueName: \"kubernetes.io/projected/2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b-kube-api-access-9x6bc\") pod \"iptables-alerter-xmtn4\" (UID: \"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b\") " pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-run-ovn\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-run-ovn-kubernetes\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120210 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-sys-fs\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84fa919-76a7-476e-9657-c195f0217bbc-tmp-dir\") pod \"node-resolver-jx6hc\" (UID: \"e84fa919-76a7-476e-9657-c195f0217bbc\") " pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120229 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-sysctl-d\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-run\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120279 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-hostroot\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/466d216d-acb1-4f99-9f98-819cc18b3367-ovnkube-config\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120335 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-registration-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-sys\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-system-cni-dir\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-systemd-units\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120429 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-slash\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.256333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-log-socket\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-var-lib-kubelet\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120501 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/36201deb-a807-451a-9a83-7446f325f7bc-konnectivity-ca\") pod \"konnectivity-agent-kxxtg\" (UID: \"36201deb-a807-451a-9a83-7446f325f7bc\") " pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120471 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-tmp\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b-iptables-alerter-script\") pod \"iptables-alerter-xmtn4\" (UID: \"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b\") " pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-cni-netd\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-etc-selinux\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-kubernetes\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/466d216d-acb1-4f99-9f98-819cc18b3367-ovnkube-script-lib\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.120668 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nv77\" (UniqueName: \"kubernetes.io/projected/7f29150c-6bc0-49cc-af95-18545831cf49-kube-api-access-7nv77\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:46.257228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.120723 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret podName:5be5526a-0c29-42b3-91fb-6c6863746390 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:46.620706997 +0000 UTC m=+1.939059274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret") pod "global-pull-secret-syncer-574vd" (UID: "5be5526a-0c29-42b3-91fb-6c6863746390") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-var-lib-cni-bin\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-run-netns\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120880 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-lib-modules\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mftf\" (UniqueName: \"kubernetes.io/projected/e84fa919-76a7-476e-9657-c195f0217bbc-kube-api-access-2mftf\") pod \"node-resolver-jx6hc\" (UID: \"e84fa919-76a7-476e-9657-c195f0217bbc\") " pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-cnibin\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.120976 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b-iptables-alerter-script\") pod \"iptables-alerter-xmtn4\" (UID: \"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b\") " pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/466d216d-acb1-4f99-9f98-819cc18b3367-ovn-node-metrics-cert\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-node-log\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswgq\" (UniqueName: \"kubernetes.io/projected/0474b40b-5711-4a08-a644-f9caf4afed37-kube-api-access-xswgq\") pod \"node-ca-cmrt4\" (UID: \"0474b40b-5711-4a08-a644-f9caf4afed37\") " pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-run-netns\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.257825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-var-lib-kubelet\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121159 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttvh\" (UniqueName: \"kubernetes.io/projected/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-kube-api-access-7ttvh\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-cni-bin\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-device-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-run-k8s-cni-cncf-io\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-var-lib-openvswitch\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121265 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqkf\" (UniqueName: \"kubernetes.io/projected/466d216d-acb1-4f99-9f98-819cc18b3367-kube-api-access-fcqkf\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121282 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-sysconfig\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-host\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-tuned\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-os-release\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121341 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m48z\" (UniqueName: \"kubernetes.io/projected/1bad4f19-6640-4c47-8340-af3bbaf3b736-kube-api-access-5m48z\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121356 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-var-lib-cni-multus\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-daemon-config\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.259022 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-run-openvswitch\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5be5526a-0c29-42b3-91fb-6c6863746390-kubelet-config\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b-host-slash\") pod \"iptables-alerter-xmtn4\" (UID: \"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b\") " pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121484 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b-host-slash\") pod \"iptables-alerter-xmtn4\" (UID: \"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b\") " pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121489 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5be5526a-0c29-42b3-91fb-6c6863746390-kubelet-config\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36201deb-a807-451a-9a83-7446f325f7bc-agent-certs\") pod \"konnectivity-agent-kxxtg\" (UID: \"36201deb-a807-451a-9a83-7446f325f7bc\") " pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121521 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1bad4f19-6640-4c47-8340-af3bbaf3b736-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0474b40b-5711-4a08-a644-f9caf4afed37-serviceca\") pod \"node-ca-cmrt4\" (UID: \"0474b40b-5711-4a08-a644-f9caf4afed37\") " pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-cni-dir\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-cnibin\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-run-multus-certs\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-etc-openvswitch\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/466d216d-acb1-4f99-9f98-819cc18b3367-env-overrides\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prcmb\" (UniqueName: \"kubernetes.io/projected/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-kube-api-access-prcmb\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.259824 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5be5526a-0c29-42b3-91fb-6c6863746390-dbus\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121675 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1bad4f19-6640-4c47-8340-af3bbaf3b736-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-cni-binary-copy\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-sysctl-conf\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121714 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121767 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e84fa919-76a7-476e-9657-c195f0217bbc-hosts-file\") pod \"node-resolver-jx6hc\" (UID: \"e84fa919-76a7-476e-9657-c195f0217bbc\") " pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5be5526a-0c29-42b3-91fb-6c6863746390-dbus\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.121734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e84fa919-76a7-476e-9657-c195f0217bbc-hosts-file\") pod \"node-resolver-jx6hc\" (UID: \"e84fa919-76a7-476e-9657-c195f0217bbc\") " pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.122002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.122017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-system-cni-dir\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.122019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0474b40b-5711-4a08-a644-f9caf4afed37-serviceca\") pod \"node-ca-cmrt4\" (UID: \"0474b40b-5711-4a08-a644-f9caf4afed37\") " pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.122047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c82t\" (UniqueName: \"kubernetes.io/projected/8d56988d-79db-44ef-9a76-bbbf70562699-kube-api-access-8c82t\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.122068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-modprobe-d\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.122080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-systemd\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.260592 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.122089 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.122128 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs podName:7f29150c-6bc0-49cc-af95-18545831cf49 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:46.622118541 +0000 UTC m=+1.940470806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs") pod "network-metrics-daemon-wpvj7" (UID: "7f29150c-6bc0-49cc-af95-18545831cf49") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.124477 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/36201deb-a807-451a-9a83-7446f325f7bc-agent-certs\") pod \"konnectivity-agent-kxxtg\" (UID: \"36201deb-a807-451a-9a83-7446f325f7bc\") " pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.130226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswgq\" (UniqueName: \"kubernetes.io/projected/0474b40b-5711-4a08-a644-f9caf4afed37-kube-api-access-xswgq\") pod \"node-ca-cmrt4\" (UID: \"0474b40b-5711-4a08-a644-f9caf4afed37\") " pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.130392 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.130405 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.130414 2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7mxf for pod openshift-network-diagnostics/network-check-target-ff7jr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.130468 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf podName:a489a534-0ee3-4d50-9bd3-d77da326f316 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:46.630437241 +0000 UTC m=+1.948789504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k7mxf" (UniqueName: "kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf") pod "network-check-target-ff7jr" (UID: "a489a534-0ee3-4d50-9bd3-d77da326f316") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.132400 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mftf\" (UniqueName: \"kubernetes.io/projected/e84fa919-76a7-476e-9657-c195f0217bbc-kube-api-access-2mftf\") pod \"node-resolver-jx6hc\" (UID: \"e84fa919-76a7-476e-9657-c195f0217bbc\") " pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.133332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6bc\" (UniqueName: \"kubernetes.io/projected/2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b-kube-api-access-9x6bc\") pod \"iptables-alerter-xmtn4\" (UID: \"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b\") " pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.135195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nv77\" (UniqueName: \"kubernetes.io/projected/7f29150c-6bc0-49cc-af95-18545831cf49-kube-api-access-7nv77\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.158671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xmtn4" Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222516 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-var-lib-cni-bin\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.261171 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222541 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-run-netns\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-lib-modules\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-cnibin\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/466d216d-acb1-4f99-9f98-819cc18b3367-ovn-node-metrics-cert\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-node-log\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-run-netns\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-var-lib-kubelet\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222661 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttvh\" (UniqueName: \"kubernetes.io/projected/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-kube-api-access-7ttvh\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-cni-bin\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-device-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-run-k8s-cni-cncf-io\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222742 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-var-lib-openvswitch\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222761 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqkf\" (UniqueName: \"kubernetes.io/projected/466d216d-acb1-4f99-9f98-819cc18b3367-kube-api-access-fcqkf\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.261675 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-sysconfig\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-host\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-tuned\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-os-release\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m48z\" (UniqueName: \"kubernetes.io/projected/1bad4f19-6640-4c47-8340-af3bbaf3b736-kube-api-access-5m48z\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-var-lib-cni-multus\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-daemon-config\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-run-openvswitch\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1bad4f19-6640-4c47-8340-af3bbaf3b736-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-cni-dir\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-cnibin\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-run-multus-certs\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-etc-openvswitch\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.222993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/466d216d-acb1-4f99-9f98-819cc18b3367-env-overrides\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.262241 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prcmb\" (UniqueName: \"kubernetes.io/projected/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-kube-api-access-prcmb\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1bad4f19-6640-4c47-8340-af3bbaf3b736-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-cni-binary-copy\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-sysctl-conf\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-system-cni-dir\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c82t\" (UniqueName: \"kubernetes.io/projected/8d56988d-79db-44ef-9a76-bbbf70562699-kube-api-access-8c82t\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-modprobe-d\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-systemd\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-os-release\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-socket-dir-parent\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-socket-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bad4f19-6640-4c47-8340-af3bbaf3b736-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-conf-dir\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.262666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-etc-kubernetes\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-kubelet\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-run-systemd\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-run-ovn\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-run-ovn-kubernetes\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-sys-fs\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-sysctl-d\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-run\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-hostroot\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/466d216d-acb1-4f99-9f98-819cc18b3367-ovnkube-config\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-registration-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-sys\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-system-cni-dir\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-systemd-units\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-slash\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263150 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-log-socket\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-var-lib-kubelet\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-tmp\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223623 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-cni-netd\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-etc-selinux\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-kubernetes\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.223687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/466d216d-acb1-4f99-9f98-819cc18b3367-ovnkube-script-lib\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.224175 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/466d216d-acb1-4f99-9f98-819cc18b3367-ovnkube-script-lib\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.224237 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-os-release\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.224436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-var-lib-cni-multus\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.224872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-daemon-config\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.224915 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-run-openvswitch\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225218 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-tuned\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225271 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1bad4f19-6640-4c47-8340-af3bbaf3b736-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.263681 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225318 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225326 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-cni-dir\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225351 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-run-netns\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225358 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-cnibin\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-var-lib-cni-bin\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225393 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-run-multus-certs\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225416 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-etc-openvswitch\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225417 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225541 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-lib-modules\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225772 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/466d216d-acb1-4f99-9f98-819cc18b3367-env-overrides\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-cnibin\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.225934 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1bad4f19-6640-4c47-8340-af3bbaf3b736-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-cni-binary-copy\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226276 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-sysctl-conf\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226308 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-system-cni-dir\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264044 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226523 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-modprobe-d\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226556 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-systemd\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226590 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-os-release\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226620 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-socket-dir-parent\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226644 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226707 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-socket-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.226982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bad4f19-6640-4c47-8340-af3bbaf3b736-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-multus-conf-dir\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227060 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-etc-kubernetes\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227085 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-kubelet\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-run-systemd\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227311 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-run-ovn\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227338 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-run-ovn-kubernetes\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227366 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-sys-fs\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227434 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-sysctl-d\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.264566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227486 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-run\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227518 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-hostroot\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227615 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/466d216d-acb1-4f99-9f98-819cc18b3367-ovn-node-metrics-cert\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227659 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-node-log\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-run-netns\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227707 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-var-lib-kubelet\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227804 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/466d216d-acb1-4f99-9f98-819cc18b3367-ovnkube-config\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-cni-bin\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227880 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-systemd-units\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227881 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-sys\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bad4f19-6640-4c47-8340-af3bbaf3b736-system-cni-dir\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-slash\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-device-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-log-socket\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227948 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-host-run-k8s-cni-cncf-io\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.265138 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-var-lib-kubelet\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227971 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-var-lib-openvswitch\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.228130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-sysconfig\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.228163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-host\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.227851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-registration-dir\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.228253 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/466d216d-acb1-4f99-9f98-819cc18b3367-host-cni-netd\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.228542 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d56988d-79db-44ef-9a76-bbbf70562699-etc-selinux\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.228600 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-etc-kubernetes\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.229719 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-tmp\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.233785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m48z\" (UniqueName: \"kubernetes.io/projected/1bad4f19-6640-4c47-8340-af3bbaf3b736-kube-api-access-5m48z\") pod \"multus-additional-cni-plugins-n8mn7\" (UID: \"1bad4f19-6640-4c47-8340-af3bbaf3b736\") " pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.233879 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prcmb\" (UniqueName: \"kubernetes.io/projected/3835c3c9-19ba-472f-a4db-98c9c9be4c4d-kube-api-access-prcmb\") pod \"tuned-7s78t\" (UID: \"3835c3c9-19ba-472f-a4db-98c9c9be4c4d\") " pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.234609 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c82t\" (UniqueName: \"kubernetes.io/projected/8d56988d-79db-44ef-9a76-bbbf70562699-kube-api-access-8c82t\") pod \"aws-ebs-csi-driver-node-hjqnf\" (UID: \"8d56988d-79db-44ef-9a76-bbbf70562699\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.235261 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttvh\" (UniqueName: \"kubernetes.io/projected/fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc-kube-api-access-7ttvh\") pod \"multus-vpkxf\" (UID: \"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc\") " pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.235518 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqkf\" (UniqueName: \"kubernetes.io/projected/466d216d-acb1-4f99-9f98-819cc18b3367-kube-api-access-fcqkf\") pod \"ovnkube-node-sjc24\" (UID: \"466d216d-acb1-4f99-9f98-819cc18b3367\") " pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.265722 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.264362 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd794cd41a0501a11ba5f67d42185203.slice/crio-f1d5cfb9589fb5339b7d17327a24ec889fcc90b9f1aad10352efbce959471c1b WatchSource:0}: Error finding container f1d5cfb9589fb5339b7d17327a24ec889fcc90b9f1aad10352efbce959471c1b: Status 404 returned error can't find the container with id f1d5cfb9589fb5339b7d17327a24ec889fcc90b9f1aad10352efbce959471c1b Jul 20 22:53:46.266245 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.264664 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2e386cf084133e9c2b8d9bc35bd306.slice/crio-10edc5bda975860b12e50ff83cbcbf811bae657cda79a9356fa19da52ba239b3 WatchSource:0}: Error finding container 10edc5bda975860b12e50ff83cbcbf811bae657cda79a9356fa19da52ba239b3: Status 404 returned error can't find the container with id 10edc5bda975860b12e50ff83cbcbf811bae657cda79a9356fa19da52ba239b3 Jul 20 22:53:46.266245 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.264743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vpkxf" Jul 20 22:53:46.270637 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.270609 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:53:46.271527 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.271512 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 22:53:46.274597 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.274580 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba34b3e_e583_4ff1_9e54_8fcf1d1ddedc.slice/crio-2b2bc93e875299c0c79035f417a3e6c57e5faa8600688e10c1476ff5f39e7905 WatchSource:0}: Error finding container 2b2bc93e875299c0c79035f417a3e6c57e5faa8600688e10c1476ff5f39e7905: Status 404 returned error can't find the container with id 2b2bc93e875299c0c79035f417a3e6c57e5faa8600688e10c1476ff5f39e7905 Jul 20 22:53:46.278713 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.278694 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466d216d_acb1_4f99_9f98_819cc18b3367.slice/crio-ad36e301b6dbb7b0bb83d29d1b3a656421f2fc17cbde64fbdcb77757c424810f WatchSource:0}: Error finding container ad36e301b6dbb7b0bb83d29d1b3a656421f2fc17cbde64fbdcb77757c424810f: Status 404 returned error can't find the container with id ad36e301b6dbb7b0bb83d29d1b3a656421f2fc17cbde64fbdcb77757c424810f Jul 20 22:53:46.361795 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.361708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:53:46.370238 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.370200 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36201deb_a807_451a_9a83_7446f325f7bc.slice/crio-4255373ee2288e1c09394fbe51f313508fe3e6abd73dcd0ffed04987464bd6b9 WatchSource:0}: Error finding container 4255373ee2288e1c09394fbe51f313508fe3e6abd73dcd0ffed04987464bd6b9: Status 404 returned error can't find the container with id 4255373ee2288e1c09394fbe51f313508fe3e6abd73dcd0ffed04987464bd6b9 Jul 20 22:53:46.374331 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.374312 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jx6hc" Jul 20 22:53:46.382091 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.382069 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84fa919_76a7_476e_9657_c195f0217bbc.slice/crio-8e2d3ff9f336b3de7708af094edb54f5652a4e3c6e136fcd34b5eacf80e77d0f WatchSource:0}: Error finding container 8e2d3ff9f336b3de7708af094edb54f5652a4e3c6e136fcd34b5eacf80e77d0f: Status 404 returned error can't find the container with id 8e2d3ff9f336b3de7708af094edb54f5652a4e3c6e136fcd34b5eacf80e77d0f Jul 20 22:53:46.399098 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.399075 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cmrt4" Jul 20 22:53:46.406438 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.406415 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0474b40b_5711_4a08_a644_f9caf4afed37.slice/crio-cd6e545a9fbc53565b0e81ec909dfd5f874dd35bfd90798e2d5dd13823491c25 WatchSource:0}: Error finding container cd6e545a9fbc53565b0e81ec909dfd5f874dd35bfd90798e2d5dd13823491c25: Status 404 returned error can't find the container with id cd6e545a9fbc53565b0e81ec909dfd5f874dd35bfd90798e2d5dd13823491c25 Jul 20 22:53:46.475020 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.474995 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" Jul 20 22:53:46.477565 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.477536 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2daa6ff5_f330_4a9d_bdc3_0bec3c3e742b.slice/crio-19e95c6219438a21318cc407bbbdf5b694aadc851ddb874be2e343eef74dd35e WatchSource:0}: Error finding container 19e95c6219438a21318cc407bbbdf5b694aadc851ddb874be2e343eef74dd35e: Status 404 returned error can't find the container with id 19e95c6219438a21318cc407bbbdf5b694aadc851ddb874be2e343eef74dd35e Jul 20 22:53:46.482855 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.482835 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d56988d_79db_44ef_9a76_bbbf70562699.slice/crio-b80fc473ac4303aec7b460ce794776131e2723a6abd8c9a0b1d5d0c0281f8985 WatchSource:0}: Error finding container b80fc473ac4303aec7b460ce794776131e2723a6abd8c9a0b1d5d0c0281f8985: Status 404 returned error can't find the container with id b80fc473ac4303aec7b460ce794776131e2723a6abd8c9a0b1d5d0c0281f8985 Jul 20 22:53:46.492743 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.492724 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7s78t" Jul 20 22:53:46.500865 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.500834 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3835c3c9_19ba_472f_a4db_98c9c9be4c4d.slice/crio-0fb1a9c4d8f8096789253356416951105cdb82ef6e4ae1169fca28ecc100ad8f WatchSource:0}: Error finding container 0fb1a9c4d8f8096789253356416951105cdb82ef6e4ae1169fca28ecc100ad8f: Status 404 returned error can't find the container with id 0fb1a9c4d8f8096789253356416951105cdb82ef6e4ae1169fca28ecc100ad8f Jul 20 22:53:46.529414 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.529387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" Jul 20 22:53:46.538129 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:53:46.538103 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bad4f19_6640_4c47_8340_af3bbaf3b736.slice/crio-85fc72146887c411f08084fab978f43e4ae191ebab1af8f7597e90a3ac1a048a WatchSource:0}: Error finding container 85fc72146887c411f08084fab978f43e4ae191ebab1af8f7597e90a3ac1a048a: Status 404 returned error can't find the container with id 85fc72146887c411f08084fab978f43e4ae191ebab1af8f7597e90a3ac1a048a Jul 20 22:53:46.626724 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.626644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:46.626724 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.626687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:46.626883 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.626778 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:46.626883 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.626780 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:46.626883 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.626826 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret podName:5be5526a-0c29-42b3-91fb-6c6863746390 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:47.626814141 +0000 UTC m=+2.945166403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret") pod "global-pull-secret-syncer-574vd" (UID: "5be5526a-0c29-42b3-91fb-6c6863746390") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:46.626883 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.626838 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs podName:7f29150c-6bc0-49cc-af95-18545831cf49 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:47.626833092 +0000 UTC m=+2.945185354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs") pod "network-metrics-daemon-wpvj7" (UID: "7f29150c-6bc0-49cc-af95-18545831cf49") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:46.727983 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.727567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:46.727983 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.727850 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:46.727983 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.727890 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:46.727983 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.727904 2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7mxf for pod openshift-network-diagnostics/network-check-target-ff7jr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:46.728288 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:46.728016 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf podName:a489a534-0ee3-4d50-9bd3-d77da326f316 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:47.727972967 +0000 UTC m=+3.046325232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7mxf" (UniqueName: "kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf") pod "network-check-target-ff7jr" (UID: "a489a534-0ee3-4d50-9bd3-d77da326f316") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:46.842374 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:46.842155 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:47.056831 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.056605 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 22:48:45 +0000 UTC" deadline="2028-04-14 02:19:31.644144228 +0000 UTC" Jul 20 22:53:47.056831 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.056666 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15195h25m44.587481991s" Jul 20 22:53:47.223316 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.223279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:47.223507 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:47.223419 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:53:47.254960 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.254889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7s78t" event={"ID":"3835c3c9-19ba-472f-a4db-98c9c9be4c4d","Type":"ContainerStarted","Data":"0fb1a9c4d8f8096789253356416951105cdb82ef6e4ae1169fca28ecc100ad8f"} Jul 20 22:53:47.265475 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.265421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" event={"ID":"8d56988d-79db-44ef-9a76-bbbf70562699","Type":"ContainerStarted","Data":"b80fc473ac4303aec7b460ce794776131e2723a6abd8c9a0b1d5d0c0281f8985"} Jul 20 22:53:47.288928 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.288889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xmtn4" event={"ID":"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b","Type":"ContainerStarted","Data":"19e95c6219438a21318cc407bbbdf5b694aadc851ddb874be2e343eef74dd35e"} Jul 20 22:53:47.301943 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.301904 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cmrt4" event={"ID":"0474b40b-5711-4a08-a644-f9caf4afed37","Type":"ContainerStarted","Data":"cd6e545a9fbc53565b0e81ec909dfd5f874dd35bfd90798e2d5dd13823491c25"} Jul 20 22:53:47.335235 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.335104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"ad36e301b6dbb7b0bb83d29d1b3a656421f2fc17cbde64fbdcb77757c424810f"} Jul 20 22:53:47.346865 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.346819 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vpkxf" event={"ID":"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc","Type":"ContainerStarted","Data":"2b2bc93e875299c0c79035f417a3e6c57e5faa8600688e10c1476ff5f39e7905"} Jul 20 22:53:47.364783 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.364746 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jx6hc" event={"ID":"e84fa919-76a7-476e-9657-c195f0217bbc","Type":"ContainerStarted","Data":"8e2d3ff9f336b3de7708af094edb54f5652a4e3c6e136fcd34b5eacf80e77d0f"} Jul 20 22:53:47.390704 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.390664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kxxtg" event={"ID":"36201deb-a807-451a-9a83-7446f325f7bc","Type":"ContainerStarted","Data":"4255373ee2288e1c09394fbe51f313508fe3e6abd73dcd0ffed04987464bd6b9"} Jul 20 22:53:47.415132 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.415094 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" event={"ID":"bd794cd41a0501a11ba5f67d42185203","Type":"ContainerStarted","Data":"f1d5cfb9589fb5339b7d17327a24ec889fcc90b9f1aad10352efbce959471c1b"} Jul 20 22:53:47.438365 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.438217 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal" event={"ID":"4f2e386cf084133e9c2b8d9bc35bd306","Type":"ContainerStarted","Data":"10edc5bda975860b12e50ff83cbcbf811bae657cda79a9356fa19da52ba239b3"} Jul 20 22:53:47.464833 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.464793 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" event={"ID":"1bad4f19-6640-4c47-8340-af3bbaf3b736","Type":"ContainerStarted","Data":"85fc72146887c411f08084fab978f43e4ae191ebab1af8f7597e90a3ac1a048a"} Jul 20 22:53:47.637621 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.637434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:47.637621 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.637539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:47.637834 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:47.637690 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:47.637834 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:47.637754 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs podName:7f29150c-6bc0-49cc-af95-18545831cf49 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:49.63773626 +0000 UTC m=+4.956088531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs") pod "network-metrics-daemon-wpvj7" (UID: "7f29150c-6bc0-49cc-af95-18545831cf49") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:47.638348 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:47.638328 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:47.638448 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:47.638390 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret podName:5be5526a-0c29-42b3-91fb-6c6863746390 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:49.638374149 +0000 UTC m=+4.956726417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret") pod "global-pull-secret-syncer-574vd" (UID: "5be5526a-0c29-42b3-91fb-6c6863746390") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:47.738901 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.738863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:47.739094 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:47.739071 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:47.739156 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:47.739102 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:47.739156 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:47.739118 2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7mxf for pod openshift-network-diagnostics/network-check-target-ff7jr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:47.739274 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:47.739181 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf podName:a489a534-0ee3-4d50-9bd3-d77da326f316 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:49.739160988 +0000 UTC m=+5.057513256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7mxf" (UniqueName: "kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf") pod "network-check-target-ff7jr" (UID: "a489a534-0ee3-4d50-9bd3-d77da326f316") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:47.853973 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:47.853756 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:48.057442 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:48.057056 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 22:48:45 +0000 UTC" deadline="2028-04-08 14:57:21.339457718 +0000 UTC" Jul 20 22:53:48.057442 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:48.057093 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15064h3m33.282369574s" Jul 20 22:53:48.221688 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:48.221635 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:48.221688 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:48.221673 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:48.221916 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:48.221789 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:53:48.221976 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:48.221933 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:53:49.222316 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:49.221686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:49.222316 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:49.221853 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:53:49.656413 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:49.656372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:49.656615 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:49.656446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:49.656687 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:49.656626 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:49.656746 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:49.656687 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret podName:5be5526a-0c29-42b3-91fb-6c6863746390 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:53.656669669 +0000 UTC m=+8.975021939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret") pod "global-pull-secret-syncer-574vd" (UID: "5be5526a-0c29-42b3-91fb-6c6863746390") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:49.657142 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:49.657085 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:49.657142 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:49.657138 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs podName:7f29150c-6bc0-49cc-af95-18545831cf49 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:53.657123013 +0000 UTC m=+8.975475277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs") pod "network-metrics-daemon-wpvj7" (UID: "7f29150c-6bc0-49cc-af95-18545831cf49") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:49.756907 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:49.756813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:49.757073 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:49.757001 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:49.757073 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:49.757023 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:49.757073 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:49.757037 2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7mxf for pod openshift-network-diagnostics/network-check-target-ff7jr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:49.757239 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:49.757110 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf podName:a489a534-0ee3-4d50-9bd3-d77da326f316 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:53.757090364 +0000 UTC m=+9.075442653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7mxf" (UniqueName: "kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf") pod "network-check-target-ff7jr" (UID: "a489a534-0ee3-4d50-9bd3-d77da326f316") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:50.221898 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:50.221757 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:50.221898 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:50.221893 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:53:50.222149 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:50.222093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:50.222232 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:50.222206 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:53:51.222591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:51.222553 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:51.223059 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:51.222707 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:53:52.221782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:52.221617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:52.221782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:52.221656 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:52.221782 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:52.221775 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:53:52.222046 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:52.221905 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:53:53.222166 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:53.222135 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:53.222630 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:53.222282 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:53:53.691419 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:53.691376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:53.691628 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:53.691474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:53.691628 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:53.691594 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:53.691733 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:53.691644 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:53.691733 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:53.691659 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs podName:7f29150c-6bc0-49cc-af95-18545831cf49 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:01.691643919 +0000 UTC m=+17.009996182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs") pod "network-metrics-daemon-wpvj7" (UID: "7f29150c-6bc0-49cc-af95-18545831cf49") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:53.691733 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:53.691703 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret podName:5be5526a-0c29-42b3-91fb-6c6863746390 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:01.691685537 +0000 UTC m=+17.010037801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret") pod "global-pull-secret-syncer-574vd" (UID: "5be5526a-0c29-42b3-91fb-6c6863746390") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:53.792935 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:53.792889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:53.793160 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:53.793138 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:53.793160 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:53.793160 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:53.793271 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:53.793175 2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7mxf for pod openshift-network-diagnostics/network-check-target-ff7jr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:53.793271 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:53.793236 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf podName:a489a534-0ee3-4d50-9bd3-d77da326f316 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:01.793218974 +0000 UTC m=+17.111571252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7mxf" (UniqueName: "kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf") pod "network-check-target-ff7jr" (UID: "a489a534-0ee3-4d50-9bd3-d77da326f316") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:54.222770 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:54.222225 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:54.222770 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:54.222270 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:54.222770 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:54.222378 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:53:54.222770 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:54.222558 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:53:55.228742 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:55.228686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:55.229204 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:55.228806 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:53:56.222551 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:56.222515 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:56.222748 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:56.222660 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:53:56.222748 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:56.222700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:56.222869 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:56.222781 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:53:57.222120 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:57.222083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:57.222734 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:57.222204 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:53:58.222425 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:58.222390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:53:58.222884 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:58.222541 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:53:58.222884 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:58.222572 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:53:58.222884 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:58.222690 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:53:59.221600 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:53:59.221558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:53:59.221791 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:53:59.221732 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:00.221623 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:00.221579 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:00.222089 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:00.221600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:00.222089 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:00.221745 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:00.222089 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:00.221879 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:01.221865 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:01.221833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:01.222281 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:01.221966 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:01.751555 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:01.751519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:01.751780 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:01.751580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:01.751780 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:01.751679 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:54:01.751780 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:01.751684 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:54:01.751780 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:01.751741 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret podName:5be5526a-0c29-42b3-91fb-6c6863746390 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:17.751721492 +0000 UTC m=+33.070073777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret") pod "global-pull-secret-syncer-574vd" (UID: "5be5526a-0c29-42b3-91fb-6c6863746390") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:54:01.751780 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:01.751757 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs podName:7f29150c-6bc0-49cc-af95-18545831cf49 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:17.751750279 +0000 UTC m=+33.070102543 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs") pod "network-metrics-daemon-wpvj7" (UID: "7f29150c-6bc0-49cc-af95-18545831cf49") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:54:01.852547 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:01.852508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:01.852749 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:01.852670 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:54:01.852749 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:01.852688 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:54:01.852749 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:01.852697 2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7mxf for pod openshift-network-diagnostics/network-check-target-ff7jr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:54:01.852853 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:01.852752 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf podName:a489a534-0ee3-4d50-9bd3-d77da326f316 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:17.852737876 +0000 UTC m=+33.171090157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7mxf" (UniqueName: "kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf") pod "network-check-target-ff7jr" (UID: "a489a534-0ee3-4d50-9bd3-d77da326f316") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:54:02.222695 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:02.222480 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:02.223167 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:02.222480 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:02.223167 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:02.222826 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:02.223167 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:02.222901 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:03.222186 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:03.222145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:03.222373 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:03.222304 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:04.221918 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:04.221882 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:04.222369 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:04.221928 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:04.222369 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:04.222009 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:04.222369 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:04.222161 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:05.222946 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:05.222909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:05.223444 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:05.223036 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:06.221845 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:06.221818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:06.221947 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:06.221929 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:06.222010 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:06.221986 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:06.222127 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:06.222098 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:06.509319 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:06.509290 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7s78t" event={"ID":"3835c3c9-19ba-472f-a4db-98c9c9be4c4d","Type":"ContainerStarted","Data":"2ea57b5bbbb6ad122cf5d8b9e2fb3b89785b934ebba6a5c13bcf94ec01d9954d"} Jul 20 22:54:06.514116 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:06.514088 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"c31a6991d0c53a5a31f31b0416f2151d6a7b5486adb5c1678aecc43252155db3"} Jul 20 22:54:06.514244 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:06.514228 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"679e624378f5d1b763fc75b7d154881b85c44402185fc2a566bc2370a453ea89"} Jul 20 22:54:06.515492 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:06.515447 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal" event={"ID":"4f2e386cf084133e9c2b8d9bc35bd306","Type":"ContainerStarted","Data":"c7abe189f2c40b12d377d7ef07ab60ded06f1f93a484fd9126f19626d91a6025"} Jul 20 22:54:06.527373 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:06.527329 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7s78t" podStartSLOduration=1.9576826010000001 podStartE2EDuration="21.52731559s" totalImagesPullingTime="19.569632989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:06.526884519 +0000 UTC m=+21.845236804" watchObservedRunningTime="2026-07-20 22:54:06.52731559 +0000 UTC m=+21.845667872" Jul 20 22:54:06.548218 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:06.547975 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-209.ec2.internal" podStartSLOduration=21.547951408 podStartE2EDuration="21.547951408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:06.547170927 +0000 UTC m=+21.865523218" watchObservedRunningTime="2026-07-20 22:54:06.547951408 +0000 UTC m=+21.866303695" Jul 20 22:54:07.222716 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.222543 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:07.222956 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:07.222830 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:07.518828 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.518732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kxxtg" event={"ID":"36201deb-a807-451a-9a83-7446f325f7bc","Type":"ContainerStarted","Data":"0e3048fecbe0a9e301414e4d3d7172fb7aab7fe68cfb677e679d2b6590d9735a"} Jul 20 22:54:07.520372 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.520326 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd794cd41a0501a11ba5f67d42185203" containerID="61185aef97185005459e2d25830162f1ac0b9489d93ab24799fce5f286bb7f91" exitCode=0 Jul 20 22:54:07.520512 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.520408 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" event={"ID":"bd794cd41a0501a11ba5f67d42185203","Type":"ContainerDied","Data":"61185aef97185005459e2d25830162f1ac0b9489d93ab24799fce5f286bb7f91"} Jul 20 22:54:07.522273 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.522086 2574 generic.go:356] "Generic (PLEG): container finished" podID="1bad4f19-6640-4c47-8340-af3bbaf3b736" containerID="476a3e6a4df6e1a0ff841d21eac585e69a6babfb89f4bd23c91bf4d9d0d3f548" exitCode=0 Jul 20 22:54:07.522384 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.522177 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" event={"ID":"1bad4f19-6640-4c47-8340-af3bbaf3b736","Type":"ContainerDied","Data":"476a3e6a4df6e1a0ff841d21eac585e69a6babfb89f4bd23c91bf4d9d0d3f548"} Jul 20 22:54:07.523972 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.523944 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" event={"ID":"8d56988d-79db-44ef-9a76-bbbf70562699","Type":"ContainerStarted","Data":"69b99d8209fb9c98e688ab020930c12995573f587e60d628dddab917e59fd3d1"} Jul 20 22:54:07.525748 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.525724 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cmrt4" event={"ID":"0474b40b-5711-4a08-a644-f9caf4afed37","Type":"ContainerStarted","Data":"fbe668ba0cec614bd213a8c02bf327e0996b69738d0ed21454f1ec81bb30219f"} Jul 20 22:54:07.528987 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.528959 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 22:54:07.529330 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.529308 2574 generic.go:356] "Generic (PLEG): container finished" podID="466d216d-acb1-4f99-9f98-819cc18b3367" containerID="c31a6991d0c53a5a31f31b0416f2151d6a7b5486adb5c1678aecc43252155db3" exitCode=1 Jul 20 22:54:07.529438 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.529381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerDied","Data":"c31a6991d0c53a5a31f31b0416f2151d6a7b5486adb5c1678aecc43252155db3"} Jul 20 22:54:07.529524 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.529486 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"12d3ad64e8f276414ba068001844e863e6ab43dd46e3e16478c03aa3e854e7ed"} Jul 20 22:54:07.529582 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.529541 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"f7fc3fc508c045b6f50c960d2ef6f6b0032504baef0cc3ddfe9153c6ba8e06f8"} Jul 20 22:54:07.529640 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.529592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"31e4a33a6c38c0a85425e2d67d809313200bb740f91c8fac442cb7c0dbb5f38a"} Jul 20 22:54:07.529694 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.529643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"207aceefddee9768bc2da0d89409ee51feb767b32aa302e54f4553946014c7a2"} Jul 20 22:54:07.531016 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.530992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vpkxf" event={"ID":"fba34b3e-e583-4ff1-9e54-8fcf1d1ddedc","Type":"ContainerStarted","Data":"caa2837b8bb1cbe2b0db4c07cd45c32b3f971cb5ffe484bad90a6dbb6b96b64a"} Jul 20 22:54:07.532567 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.532520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jx6hc" event={"ID":"e84fa919-76a7-476e-9657-c195f0217bbc","Type":"ContainerStarted","Data":"0bc7009bd5b83c147c7fe9297a33a6802bf5b41c19f32ceadf12dbd066fb27ca"} Jul 20 22:54:07.535719 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.535670 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kxxtg" podStartSLOduration=2.854294497 podStartE2EDuration="22.53565128s" totalImagesPullingTime="19.681356783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:07.534539761 +0000 UTC m=+22.852892050" watchObservedRunningTime="2026-07-20 22:54:07.53565128 +0000 UTC m=+22.854003566" Jul 20 22:54:07.600448 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.600395 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vpkxf" podStartSLOduration=2.290967323 podStartE2EDuration="22.600372196s" totalImagesPullingTime="20.309404873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:07.599511142 +0000 UTC m=+22.917863428" watchObservedRunningTime="2026-07-20 22:54:07.600372196 +0000 UTC m=+22.918724486" Jul 20 22:54:07.617146 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.617102 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jx6hc" podStartSLOduration=2.931004224 podStartE2EDuration="22.617086306s" totalImagesPullingTime="19.686082082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:07.616519857 +0000 UTC m=+22.934872148" watchObservedRunningTime="2026-07-20 22:54:07.617086306 +0000 UTC m=+22.935438591" Jul 20 22:54:07.632697 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.632655 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cmrt4" podStartSLOduration=2.970602927 podStartE2EDuration="22.632638592s" totalImagesPullingTime="19.662035665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:07.632026308 +0000 UTC m=+22.950378595" watchObservedRunningTime="2026-07-20 22:54:07.632638592 +0000 UTC m=+22.950990918" Jul 20 22:54:07.760253 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.760221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxtg_36201deb-a807-451a-9a83-7446f325f7bc/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:07.835602 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:07.835566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxtg_36201deb-a807-451a-9a83-7446f325f7bc/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:08.064368 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.064341 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 22:54:08.114863 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.114740 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T22:54:08.06435964Z","UUID":"1b02c4a5-a014-4f5a-9b72-bab76ce1caa3","Handler":null,"Name":"","Endpoint":""} Jul 20 22:54:08.116507 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.116476 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 22:54:08.116666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.116521 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 22:54:08.221545 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.221500 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:08.221545 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.221543 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:08.221782 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:08.221631 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:08.221782 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:08.221769 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:08.537065 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.537024 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" event={"ID":"8d56988d-79db-44ef-9a76-bbbf70562699","Type":"ContainerStarted","Data":"b1353573691b837c8b1c00da91daf5a46b4a25106c96aa209adbda4524f03e5f"} Jul 20 22:54:08.538782 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.538746 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xmtn4" event={"ID":"2daa6ff5-f330-4a9d-bdc3-0bec3c3e742b","Type":"ContainerStarted","Data":"b9ee43f4f33c2a91f998ffe0e70b069e3d0e08bf5281b43fbf51ff33e4a806e1"} Jul 20 22:54:08.540758 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.540735 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" event={"ID":"bd794cd41a0501a11ba5f67d42185203","Type":"ContainerStarted","Data":"b76215f852f6f7e5e73c67a45af255b541ffcb2f7616404dfdbe2dc92a39476b"} Jul 20 22:54:08.554890 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.554833 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xmtn4" podStartSLOduration=3.964727231 podStartE2EDuration="23.554814061s" totalImagesPullingTime="19.59008683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:08.55460818 +0000 UTC m=+23.872960465" watchObservedRunningTime="2026-07-20 22:54:08.554814061 +0000 UTC m=+23.873166329" Jul 20 22:54:08.570292 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.570228 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-209.ec2.internal" podStartSLOduration=23.57020514 podStartE2EDuration="23.57020514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:08.569571124 +0000 UTC m=+23.887923412" watchObservedRunningTime="2026-07-20 22:54:08.57020514 +0000 UTC m=+23.888557427" Jul 20 22:54:08.693039 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.693000 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxtg_36201deb-a807-451a-9a83-7446f325f7bc/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:08.775336 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.775296 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxtg_36201deb-a807-451a-9a83-7446f325f7bc/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:08.838499 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:08.838454 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxtg_36201deb-a807-451a-9a83-7446f325f7bc/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:09.222159 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:09.221844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:09.222159 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:09.222095 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:09.518105 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:09.518027 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:54:09.518841 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:09.518616 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:54:09.546187 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:09.546168 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 22:54:09.546603 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:09.546536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"2f172c59f564f4fdff33573ed1d50b8bcb52cb07be1aeec9fa1f225d29435de9"} Jul 20 22:54:09.547045 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:09.547025 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:54:09.547291 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:09.547268 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kxxtg" Jul 20 22:54:09.611084 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:09.610958 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxtg_36201deb-a807-451a-9a83-7446f325f7bc/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:10.221807 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:10.221768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:10.222005 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:10.221778 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:10.222005 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:10.221900 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:10.222005 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:10.221976 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:10.422144 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:10.422109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxtg_36201deb-a807-451a-9a83-7446f325f7bc/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:10.472360 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:10.472259 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kxxtg_36201deb-a807-451a-9a83-7446f325f7bc/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:10.551625 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:10.551587 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" event={"ID":"8d56988d-79db-44ef-9a76-bbbf70562699","Type":"ContainerStarted","Data":"5d0c3d1f50a1ccd6066257ae6a38054b95eb7cdb3468bc9c51240924b9ca76dc"} Jul 20 22:54:10.570901 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:10.570842 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjqnf" podStartSLOduration=3.467831023 podStartE2EDuration="25.570820838s" totalImagesPullingTime="22.102989815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:10.570589484 +0000 UTC m=+25.888941771" watchObservedRunningTime="2026-07-20 22:54:10.570820838 +0000 UTC m=+25.889173125" Jul 20 22:54:11.222496 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:11.222452 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:11.222638 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:11.222613 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:12.222650 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:12.222283 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:12.222650 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:12.222330 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:12.222650 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:12.222631 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:12.223517 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:12.222699 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:12.560422 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:12.560387 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 22:54:12.560830 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:12.560801 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"22e52f522cda9492d05cd2e4bf73def243b6b8c6d6d6f032c1422c0174a68cb1"} Jul 20 22:54:12.561056 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:12.561036 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:54:12.561275 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:12.561256 2574 scope.go:117] "RemoveContainer" containerID="c31a6991d0c53a5a31f31b0416f2151d6a7b5486adb5c1678aecc43252155db3" Jul 20 22:54:12.563221 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:12.563195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" event={"ID":"1bad4f19-6640-4c47-8340-af3bbaf3b736","Type":"ContainerStarted","Data":"ae18b21708be859e65276ba95af07916723ef63655d93d62d92c3840e1350d73"} Jul 20 22:54:12.578029 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:12.578001 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:54:13.222456 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:13.222423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:13.222635 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:13.222553 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:13.567897 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:13.567720 2574 generic.go:356] "Generic (PLEG): container finished" podID="1bad4f19-6640-4c47-8340-af3bbaf3b736" containerID="ae18b21708be859e65276ba95af07916723ef63655d93d62d92c3840e1350d73" exitCode=0 Jul 20 22:54:13.568638 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:13.567804 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" event={"ID":"1bad4f19-6640-4c47-8340-af3bbaf3b736","Type":"ContainerDied","Data":"ae18b21708be859e65276ba95af07916723ef63655d93d62d92c3840e1350d73"} Jul 20 22:54:13.574890 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:13.574873 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 22:54:13.575214 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:13.575193 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" event={"ID":"466d216d-acb1-4f99-9f98-819cc18b3367","Type":"ContainerStarted","Data":"4a02944ee911792040fd615f80bf188edcfca4f5e38694a81841abbc18e7b52c"} Jul 20 22:54:13.575309 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:13.575290 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 20 22:54:13.575505 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:13.575484 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:54:13.591454 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:13.591425 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:54:13.614967 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:13.614922 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" podStartSLOduration=8.795131098 podStartE2EDuration="28.614908959s" totalImagesPullingTime="19.819777861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:13.61476289 +0000 UTC m=+28.933115175" watchObservedRunningTime="2026-07-20 22:54:13.614908959 +0000 UTC m=+28.933261244" Jul 20 22:54:14.221641 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:14.221608 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:14.221822 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:14.221732 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:14.221822 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:14.221787 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:14.221935 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:14.221911 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:14.240559 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:14.240524 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ff7jr"] Jul 20 22:54:14.243919 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:14.243891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-574vd"] Jul 20 22:54:14.245088 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:14.244632 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wpvj7"] Jul 20 22:54:14.245088 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:14.244749 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:14.245088 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:14.244865 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:14.563480 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:14.563428 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:54:14.577161 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:14.577138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:14.577703 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:14.577143 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:14.577703 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:14.577242 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:14.577703 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:14.577306 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:15.581945 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:15.581874 2574 generic.go:356] "Generic (PLEG): container finished" podID="1bad4f19-6640-4c47-8340-af3bbaf3b736" containerID="8da5ade26d20ba18369037ebdbdc58d2e25aa89198b2d079569f020b4eb7be8e" exitCode=0 Jul 20 22:54:15.582304 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:15.581957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" event={"ID":"1bad4f19-6640-4c47-8340-af3bbaf3b736","Type":"ContainerDied","Data":"8da5ade26d20ba18369037ebdbdc58d2e25aa89198b2d079569f020b4eb7be8e"} Jul 20 22:54:16.221852 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:16.221766 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:16.221852 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:16.221795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:16.221852 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:16.221798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:16.222076 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:16.221895 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:16.222076 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:16.222009 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:16.222143 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:16.222110 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:17.588838 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:17.588755 2574 generic.go:356] "Generic (PLEG): container finished" podID="1bad4f19-6640-4c47-8340-af3bbaf3b736" containerID="a614c8996a7ab8e62b31a6cd3590aa87c2296fcf55a78249fbefc00fb3205d04" exitCode=0 Jul 20 22:54:17.588838 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:17.588816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" event={"ID":"1bad4f19-6640-4c47-8340-af3bbaf3b736","Type":"ContainerDied","Data":"a614c8996a7ab8e62b31a6cd3590aa87c2296fcf55a78249fbefc00fb3205d04"} Jul 20 22:54:17.771287 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:17.771236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:17.771455 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:17.771342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:17.771455 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:17.771385 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:54:17.771601 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:17.771479 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:54:17.771601 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:17.771482 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret podName:5be5526a-0c29-42b3-91fb-6c6863746390 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:49.771447911 +0000 UTC m=+65.089800176 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret") pod "global-pull-secret-syncer-574vd" (UID: "5be5526a-0c29-42b3-91fb-6c6863746390") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:54:17.771601 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:17.771540 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs podName:7f29150c-6bc0-49cc-af95-18545831cf49 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:49.771524414 +0000 UTC m=+65.089876691 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs") pod "network-metrics-daemon-wpvj7" (UID: "7f29150c-6bc0-49cc-af95-18545831cf49") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:54:17.872589 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:17.872556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:17.872770 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:17.872702 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:54:17.872770 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:17.872719 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:54:17.872770 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:17.872732 2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7mxf for pod openshift-network-diagnostics/network-check-target-ff7jr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:54:17.872882 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:17.872786 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf podName:a489a534-0ee3-4d50-9bd3-d77da326f316 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:49.872772872 +0000 UTC m=+65.191125136 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7mxf" (UniqueName: "kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf") pod "network-check-target-ff7jr" (UID: "a489a534-0ee3-4d50-9bd3-d77da326f316") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:54:18.222499 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:18.222408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:18.222499 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:18.222408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:18.222718 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:18.222408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:18.222718 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:18.222563 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpvj7" podUID="7f29150c-6bc0-49cc-af95-18545831cf49" Jul 20 22:54:18.222718 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:18.222632 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ff7jr" podUID="a489a534-0ee3-4d50-9bd3-d77da326f316" Jul 20 22:54:18.222841 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:18.222725 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-574vd" podUID="5be5526a-0c29-42b3-91fb-6c6863746390" Jul 20 22:54:19.499616 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:19.499586 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-209.ec2.internal" event="NodeReady" Jul 20 22:54:19.499974 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:19.499731 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 22:54:20.221633 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:20.221599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:20.221822 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:20.221599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:20.221822 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:20.221599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:20.224750 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:20.224729 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 22:54:20.225260 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:20.225234 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 22:54:20.225997 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:20.225937 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 22:54:20.226126 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:20.226007 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 22:54:20.226126 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:20.226086 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d688j\"" Jul 20 22:54:20.226263 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:20.226227 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xr56f\"" Jul 20 22:54:22.602641 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:22.602611 2574 generic.go:356] "Generic (PLEG): container finished" podID="1bad4f19-6640-4c47-8340-af3bbaf3b736" containerID="0f5623c008b64f8273202f0742b0048a144d752d93d818588fc00b6d709c6eb9" exitCode=0 Jul 20 22:54:22.603190 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:22.602665 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" event={"ID":"1bad4f19-6640-4c47-8340-af3bbaf3b736","Type":"ContainerDied","Data":"0f5623c008b64f8273202f0742b0048a144d752d93d818588fc00b6d709c6eb9"} Jul 20 22:54:23.608101 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:23.608072 2574 generic.go:356] "Generic (PLEG): container finished" podID="1bad4f19-6640-4c47-8340-af3bbaf3b736" containerID="a263b2ac74cd045b5d8652f80fa218973ad7f32ccf429ef606a6357d5382d3d1" exitCode=0 Jul 20 22:54:23.608506 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:23.608109 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" event={"ID":"1bad4f19-6640-4c47-8340-af3bbaf3b736","Type":"ContainerDied","Data":"a263b2ac74cd045b5d8652f80fa218973ad7f32ccf429ef606a6357d5382d3d1"} Jul 20 22:54:24.613917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:24.613883 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" event={"ID":"1bad4f19-6640-4c47-8340-af3bbaf3b736","Type":"ContainerStarted","Data":"0f9f5c042037e7951a7b0e4bf0e4ee08e93c5c33df5351b7be0fd0334b1acd9b"} Jul 20 22:54:24.639743 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:24.639691 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n8mn7" podStartSLOduration=8.749369992 podStartE2EDuration="39.639674791s" totalImagesPullingTime="30.890304799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:24.637853719 +0000 UTC m=+39.956206005" watchObservedRunningTime="2026-07-20 22:54:24.639674791 +0000 UTC m=+39.958027076" Jul 20 22:54:39.591253 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.591216 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-655fdccb8d-bcs6g"] Jul 20 22:54:39.594048 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.594031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.596791 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.596769 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 20 22:54:39.597096 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.597076 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 20 22:54:39.597159 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.597121 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 20 22:54:39.597336 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.597323 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w7p55\"" Jul 20 22:54:39.604634 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.604612 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 20 22:54:39.609814 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.609794 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-655fdccb8d-bcs6g"] Jul 20 22:54:39.656861 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.656799 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mm8pc"] Jul 20 22:54:39.660712 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.660686 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rhwct"] Jul 20 22:54:39.660868 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.660828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.663634 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.663613 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c"] Jul 20 22:54:39.663748 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.663735 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rhwct" Jul 20 22:54:39.666720 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.666701 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp"] Jul 20 22:54:39.666964 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.666834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.667309 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.667227 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 22:54:39.667309 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.667260 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 22:54:39.667483 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.667309 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qkglg\"" Jul 20 22:54:39.670408 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.670384 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 22:54:39.670627 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.670611 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 22:54:39.670707 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.670612 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtmm9\"" Jul 20 22:54:39.670707 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.670693 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-86f6bc7688-gdrps"] Jul 20 22:54:39.670847 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.670833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.672449 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.672431 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 22:54:39.673184 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.673166 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 22:54:39.673323 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.673303 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6p6x6\"" Jul 20 22:54:39.673998 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.673930 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl"] Jul 20 22:54:39.674060 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.674028 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.674119 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.674079 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 20 22:54:39.675182 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.675165 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 20 22:54:39.675297 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.675183 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 20 22:54:39.677013 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.676996 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.677120 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.677038 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 20 22:54:39.677120 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.677039 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 20 22:54:39.677120 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.677070 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 20 22:54:39.677814 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.677791 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 20 22:54:39.679129 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.679109 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 22:54:39.679274 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.679207 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 22:54:39.680035 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.680002 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 20 22:54:39.680182 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.680165 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 20 22:54:39.680250 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.680197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 20 22:54:39.680384 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.680050 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 20 22:54:39.680590 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.680570 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 20 22:54:39.680794 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.680774 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 22:54:39.680959 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.680936 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 20 22:54:39.680959 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.680953 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 20 22:54:39.681228 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.681209 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x7ldt\"" Jul 20 22:54:39.690737 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.690715 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mm8pc"] Jul 20 22:54:39.696646 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.696443 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl"] Jul 20 22:54:39.697114 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.697090 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp"] Jul 20 22:54:39.697991 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.697974 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c"] Jul 20 22:54:39.698796 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.698766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86f6bc7688-gdrps"] Jul 20 22:54:39.711322 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.711298 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rhwct"] Jul 20 22:54:39.725837 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.725807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-installation-pull-secrets\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.725951 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.725850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-registry-tls\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.725951 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.725874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-image-registry-private-configuration\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.725951 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.725895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-registry-certificates\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.725951 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.725946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-bound-sa-token\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.726097 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.725964 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvk67\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-kube-api-access-wvk67\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.726097 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.725981 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dcbf07b6-9d17-4014-a290-0da81107815e-ca-trust-extracted\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.726097 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.726000 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-trusted-ca\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.735013 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.734989 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-655fdccb8d-bcs6g"] Jul 20 22:54:39.735161 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:39.735143 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-wvk67 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" podUID="dcbf07b6-9d17-4014-a290-0da81107815e" Jul 20 22:54:39.826434 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dcbf07b6-9d17-4014-a290-0da81107815e-ca-trust-extracted\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.826434 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-installation-pull-secrets\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.826709 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1395ccea-0612-4422-8bcc-050d534f6697-stats-auth\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.826709 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/47303d5f-832f-48f6-98ea-3aa738f34b38-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-6d87c\" (UID: \"47303d5f-832f-48f6-98ea-3aa738f34b38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.826709 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826683 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-registry-tls\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.826867 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-hub\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.826867 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5af0766-d27e-4caf-bb82-e6929c31b8f9-metrics-tls\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.826867 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-registry-certificates\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.826867 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5af0766-d27e-4caf-bb82-e6929c31b8f9-config-volume\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.826867 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4qgs\" (UniqueName: \"kubernetes.io/projected/1395ccea-0612-4422-8bcc-050d534f6697-kube-api-access-r4qgs\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.826867 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dcbf07b6-9d17-4014-a290-0da81107815e-ca-trust-extracted\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.827122 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1caef281-adf8-4780-a381-47e481388395-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.827122 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5af0766-d27e-4caf-bb82-e6929c31b8f9-tmp-dir\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.827122 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.826961 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1395ccea-0612-4422-8bcc-050d534f6697-default-certificate\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.827122 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-bound-sa-token\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.827122 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z624g\" (UniqueName: \"kubernetes.io/projected/f53f3607-9f4c-47f6-8ddf-d81d67b0e38e-kube-api-access-z624g\") pod \"klusterlet-addon-workmgr-7fc6b765d7-hqfcl\" (UID: \"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.827122 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvk67\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-kube-api-access-wvk67\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.827122 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjp6j\" (UniqueName: \"kubernetes.io/projected/e5af0766-d27e-4caf-bb82-e6929c31b8f9-kube-api-access-xjp6j\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.827122 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53f3607-9f4c-47f6-8ddf-d81d67b0e38e-tmp\") pod \"klusterlet-addon-workmgr-7fc6b765d7-hqfcl\" (UID: \"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.827122 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-trusted-ca\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.827591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47303d5f-832f-48f6-98ea-3aa738f34b38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6d87c\" (UID: \"47303d5f-832f-48f6-98ea-3aa738f34b38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.827591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1395ccea-0612-4422-8bcc-050d534f6697-service-ca-bundle\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.827591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7febacb1-bee8-4309-9751-6e2d693f2bcc-cert\") pod \"ingress-canary-rhwct\" (UID: \"7febacb1-bee8-4309-9751-6e2d693f2bcc\") " pod="openshift-ingress-canary/ingress-canary-rhwct" Jul 20 22:54:39.827591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827283 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1395ccea-0612-4422-8bcc-050d534f6697-metrics-certs\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.827591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-image-registry-private-configuration\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.827591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827356 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcskl\" (UniqueName: \"kubernetes.io/projected/7febacb1-bee8-4309-9751-6e2d693f2bcc-kube-api-access-kcskl\") pod \"ingress-canary-rhwct\" (UID: \"7febacb1-bee8-4309-9751-6e2d693f2bcc\") " pod="openshift-ingress-canary/ingress-canary-rhwct" Jul 20 22:54:39.827591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.827591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-ca\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.827591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827564 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j622\" (UniqueName: \"kubernetes.io/projected/47303d5f-832f-48f6-98ea-3aa738f34b38-kube-api-access-6j622\") pod \"cluster-monitoring-operator-76cf8966f8-6d87c\" (UID: \"47303d5f-832f-48f6-98ea-3aa738f34b38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.828034 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pszb5\" (UniqueName: \"kubernetes.io/projected/1caef281-adf8-4780-a381-47e481388395-kube-api-access-pszb5\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.828034 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827620 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.828034 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.827657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f53f3607-9f4c-47f6-8ddf-d81d67b0e38e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fc6b765d7-hqfcl\" (UID: \"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.828286 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.828269 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-registry-certificates\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.828367 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.828340 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-trusted-ca\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.830972 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.830947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-registry-tls\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.831100 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.831026 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-image-registry-private-configuration\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.831158 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.831096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-installation-pull-secrets\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.837757 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.837736 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-bound-sa-token\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.837884 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.837855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvk67\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-kube-api-access-wvk67\") pod \"image-registry-655fdccb8d-bcs6g\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:39.853149 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.853127 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84df66f6d-lzwgx"] Jul 20 22:54:39.857621 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.857605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:39.871787 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.871763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84df66f6d-lzwgx"] Jul 20 22:54:39.928686 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5af0766-d27e-4caf-bb82-e6929c31b8f9-config-volume\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.928686 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4qgs\" (UniqueName: \"kubernetes.io/projected/1395ccea-0612-4422-8bcc-050d534f6697-kube-api-access-r4qgs\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.928931 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1caef281-adf8-4780-a381-47e481388395-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.928931 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5af0766-d27e-4caf-bb82-e6929c31b8f9-tmp-dir\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.928931 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1395ccea-0612-4422-8bcc-050d534f6697-default-certificate\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.928931 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928777 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z624g\" (UniqueName: \"kubernetes.io/projected/f53f3607-9f4c-47f6-8ddf-d81d67b0e38e-kube-api-access-z624g\") pod \"klusterlet-addon-workmgr-7fc6b765d7-hqfcl\" (UID: \"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.928931 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928796 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjp6j\" (UniqueName: \"kubernetes.io/projected/e5af0766-d27e-4caf-bb82-e6929c31b8f9-kube-api-access-xjp6j\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.928931 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53f3607-9f4c-47f6-8ddf-d81d67b0e38e-tmp\") pod \"klusterlet-addon-workmgr-7fc6b765d7-hqfcl\" (UID: \"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.929224 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47303d5f-832f-48f6-98ea-3aa738f34b38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6d87c\" (UID: \"47303d5f-832f-48f6-98ea-3aa738f34b38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.929224 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.928996 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1395ccea-0612-4422-8bcc-050d534f6697-service-ca-bundle\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.929224 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7febacb1-bee8-4309-9751-6e2d693f2bcc-cert\") pod \"ingress-canary-rhwct\" (UID: \"7febacb1-bee8-4309-9751-6e2d693f2bcc\") " pod="openshift-ingress-canary/ingress-canary-rhwct" Jul 20 22:54:39.929224 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1395ccea-0612-4422-8bcc-050d534f6697-metrics-certs\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.929224 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929073 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5af0766-d27e-4caf-bb82-e6929c31b8f9-tmp-dir\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.929224 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcskl\" (UniqueName: \"kubernetes.io/projected/7febacb1-bee8-4309-9751-6e2d693f2bcc-kube-api-access-kcskl\") pod \"ingress-canary-rhwct\" (UID: \"7febacb1-bee8-4309-9751-6e2d693f2bcc\") " pod="openshift-ingress-canary/ingress-canary-rhwct" Jul 20 22:54:39.929224 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.929224 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-ca\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.929621 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j622\" (UniqueName: \"kubernetes.io/projected/47303d5f-832f-48f6-98ea-3aa738f34b38-kube-api-access-6j622\") pod \"cluster-monitoring-operator-76cf8966f8-6d87c\" (UID: \"47303d5f-832f-48f6-98ea-3aa738f34b38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.929621 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929264 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pszb5\" (UniqueName: \"kubernetes.io/projected/1caef281-adf8-4780-a381-47e481388395-kube-api-access-pszb5\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.929621 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.929621 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929296 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5af0766-d27e-4caf-bb82-e6929c31b8f9-config-volume\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.929621 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f53f3607-9f4c-47f6-8ddf-d81d67b0e38e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fc6b765d7-hqfcl\" (UID: \"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.929621 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53f3607-9f4c-47f6-8ddf-d81d67b0e38e-tmp\") pod \"klusterlet-addon-workmgr-7fc6b765d7-hqfcl\" (UID: \"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.929917 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.929664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1caef281-adf8-4780-a381-47e481388395-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.930314 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.930289 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1395ccea-0612-4422-8bcc-050d534f6697-service-ca-bundle\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.930479 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.930408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1395ccea-0612-4422-8bcc-050d534f6697-stats-auth\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.930611 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.930594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/47303d5f-832f-48f6-98ea-3aa738f34b38-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-6d87c\" (UID: \"47303d5f-832f-48f6-98ea-3aa738f34b38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.930735 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.930716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-hub\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.930846 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.930831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5af0766-d27e-4caf-bb82-e6929c31b8f9-metrics-tls\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.931515 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.931486 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/47303d5f-832f-48f6-98ea-3aa738f34b38-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-6d87c\" (UID: \"47303d5f-832f-48f6-98ea-3aa738f34b38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.932510 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.932250 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47303d5f-832f-48f6-98ea-3aa738f34b38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-6d87c\" (UID: \"47303d5f-832f-48f6-98ea-3aa738f34b38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.932617 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.932506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-ca\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.932904 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.932879 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f53f3607-9f4c-47f6-8ddf-d81d67b0e38e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fc6b765d7-hqfcl\" (UID: \"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.933198 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.933139 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1395ccea-0612-4422-8bcc-050d534f6697-default-certificate\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.933562 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.933527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1395ccea-0612-4422-8bcc-050d534f6697-metrics-certs\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.933666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.933586 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7febacb1-bee8-4309-9751-6e2d693f2bcc-cert\") pod \"ingress-canary-rhwct\" (UID: \"7febacb1-bee8-4309-9751-6e2d693f2bcc\") " pod="openshift-ingress-canary/ingress-canary-rhwct" Jul 20 22:54:39.934035 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.934016 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.934123 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.934042 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1395ccea-0612-4422-8bcc-050d534f6697-stats-auth\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.934163 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.934147 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-hub\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.934398 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.934379 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5af0766-d27e-4caf-bb82-e6929c31b8f9-metrics-tls\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.935132 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.935116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1caef281-adf8-4780-a381-47e481388395-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.947647 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.947625 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjp6j\" (UniqueName: \"kubernetes.io/projected/e5af0766-d27e-4caf-bb82-e6929c31b8f9-kube-api-access-xjp6j\") pod \"dns-default-mm8pc\" (UID: \"e5af0766-d27e-4caf-bb82-e6929c31b8f9\") " pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.954878 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.954854 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pszb5\" (UniqueName: \"kubernetes.io/projected/1caef281-adf8-4780-a381-47e481388395-kube-api-access-pszb5\") pod \"cluster-proxy-proxy-agent-67f85965b9-sqglp\" (UID: \"1caef281-adf8-4780-a381-47e481388395\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:39.957629 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.957611 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j622\" (UniqueName: \"kubernetes.io/projected/47303d5f-832f-48f6-98ea-3aa738f34b38-kube-api-access-6j622\") pod \"cluster-monitoring-operator-76cf8966f8-6d87c\" (UID: \"47303d5f-832f-48f6-98ea-3aa738f34b38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.959051 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.959034 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4qgs\" (UniqueName: \"kubernetes.io/projected/1395ccea-0612-4422-8bcc-050d534f6697-kube-api-access-r4qgs\") pod \"router-default-86f6bc7688-gdrps\" (UID: \"1395ccea-0612-4422-8bcc-050d534f6697\") " pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:39.961694 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.961671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcskl\" (UniqueName: \"kubernetes.io/projected/7febacb1-bee8-4309-9751-6e2d693f2bcc-kube-api-access-kcskl\") pod \"ingress-canary-rhwct\" (UID: \"7febacb1-bee8-4309-9751-6e2d693f2bcc\") " pod="openshift-ingress-canary/ingress-canary-rhwct" Jul 20 22:54:39.961768 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.961706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z624g\" (UniqueName: \"kubernetes.io/projected/f53f3607-9f4c-47f6-8ddf-d81d67b0e38e-kube-api-access-z624g\") pod \"klusterlet-addon-workmgr-7fc6b765d7-hqfcl\" (UID: \"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:39.970432 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.970417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:39.976950 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.976936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rhwct" Jul 20 22:54:39.984884 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.984859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" Jul 20 22:54:39.998004 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:39.997978 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" Jul 20 22:54:40.005256 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.005204 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:40.009839 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.009746 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:40.031488 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.031434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/359ef3fc-983f-4e37-982c-76b9ffb1adb5-installation-pull-secrets\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.031639 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.031505 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/359ef3fc-983f-4e37-982c-76b9ffb1adb5-bound-sa-token\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.031639 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.031568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmq7s\" (UniqueName: \"kubernetes.io/projected/359ef3fc-983f-4e37-982c-76b9ffb1adb5-kube-api-access-zmq7s\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.031639 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.031591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/359ef3fc-983f-4e37-982c-76b9ffb1adb5-ca-trust-extracted\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.031639 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.031616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/359ef3fc-983f-4e37-982c-76b9ffb1adb5-registry-certificates\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.031639 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.031638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/359ef3fc-983f-4e37-982c-76b9ffb1adb5-trusted-ca\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.031897 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.031691 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/359ef3fc-983f-4e37-982c-76b9ffb1adb5-registry-tls\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.031897 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.031720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/359ef3fc-983f-4e37-982c-76b9ffb1adb5-image-registry-private-configuration\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.133250 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.133197 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/359ef3fc-983f-4e37-982c-76b9ffb1adb5-bound-sa-token\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.133250 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.133250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmq7s\" (UniqueName: \"kubernetes.io/projected/359ef3fc-983f-4e37-982c-76b9ffb1adb5-kube-api-access-zmq7s\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.133489 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.133268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/359ef3fc-983f-4e37-982c-76b9ffb1adb5-ca-trust-extracted\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.133489 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.133290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/359ef3fc-983f-4e37-982c-76b9ffb1adb5-registry-certificates\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.133489 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.133310 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/359ef3fc-983f-4e37-982c-76b9ffb1adb5-trusted-ca\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.133489 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.133368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/359ef3fc-983f-4e37-982c-76b9ffb1adb5-registry-tls\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.133489 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.133397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/359ef3fc-983f-4e37-982c-76b9ffb1adb5-image-registry-private-configuration\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.133725 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.133504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/359ef3fc-983f-4e37-982c-76b9ffb1adb5-installation-pull-secrets\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.134179 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.134135 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/359ef3fc-983f-4e37-982c-76b9ffb1adb5-ca-trust-extracted\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.135451 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.135112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/359ef3fc-983f-4e37-982c-76b9ffb1adb5-registry-certificates\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.136492 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.135762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/359ef3fc-983f-4e37-982c-76b9ffb1adb5-trusted-ca\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.139284 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.138685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/359ef3fc-983f-4e37-982c-76b9ffb1adb5-registry-tls\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.139284 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.138969 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/359ef3fc-983f-4e37-982c-76b9ffb1adb5-installation-pull-secrets\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.139284 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.139086 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/359ef3fc-983f-4e37-982c-76b9ffb1adb5-image-registry-private-configuration\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.148975 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.148889 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmq7s\" (UniqueName: \"kubernetes.io/projected/359ef3fc-983f-4e37-982c-76b9ffb1adb5-kube-api-access-zmq7s\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.149283 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.149216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/359ef3fc-983f-4e37-982c-76b9ffb1adb5-bound-sa-token\") pod \"image-registry-84df66f6d-lzwgx\" (UID: \"359ef3fc-983f-4e37-982c-76b9ffb1adb5\") " pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.166518 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.166166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.326010 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.325977 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84df66f6d-lzwgx"] Jul 20 22:54:40.329429 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:40.329404 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359ef3fc_983f_4e37_982c_76b9ffb1adb5.slice/crio-894d8772e8a01cde18aeb8f38be644e8185117c02c351a5025c550b86aa7bf71 WatchSource:0}: Error finding container 894d8772e8a01cde18aeb8f38be644e8185117c02c351a5025c550b86aa7bf71: Status 404 returned error can't find the container with id 894d8772e8a01cde18aeb8f38be644e8185117c02c351a5025c550b86aa7bf71 Jul 20 22:54:40.418366 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.418260 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c"] Jul 20 22:54:40.421839 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.421776 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp"] Jul 20 22:54:40.423279 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.422914 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rhwct"] Jul 20 22:54:40.423869 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.423848 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mm8pc"] Jul 20 22:54:40.425447 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:40.425046 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1caef281_adf8_4780_a381_47e481388395.slice/crio-e0251ec785bf1be00e915e4bd4a0e43ccb47757133dceac3da659679b24aff19 WatchSource:0}: Error finding container e0251ec785bf1be00e915e4bd4a0e43ccb47757133dceac3da659679b24aff19: Status 404 returned error can't find the container with id e0251ec785bf1be00e915e4bd4a0e43ccb47757133dceac3da659679b24aff19 Jul 20 22:54:40.425894 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:40.425874 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7febacb1_bee8_4309_9751_6e2d693f2bcc.slice/crio-1da20f52266a4b0690b763551eee2b15fc46ba91a1153bde064153f9a024ba15 WatchSource:0}: Error finding container 1da20f52266a4b0690b763551eee2b15fc46ba91a1153bde064153f9a024ba15: Status 404 returned error can't find the container with id 1da20f52266a4b0690b763551eee2b15fc46ba91a1153bde064153f9a024ba15 Jul 20 22:54:40.427239 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:40.427217 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5af0766_d27e_4caf_bb82_e6929c31b8f9.slice/crio-979bfe64e76c0f91bc6b2adc459682f10d6c2b939753da9d63aa336fc85f0b84 WatchSource:0}: Error finding container 979bfe64e76c0f91bc6b2adc459682f10d6c2b939753da9d63aa336fc85f0b84: Status 404 returned error can't find the container with id 979bfe64e76c0f91bc6b2adc459682f10d6c2b939753da9d63aa336fc85f0b84 Jul 20 22:54:40.465591 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.465563 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl"] Jul 20 22:54:40.468252 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:40.468221 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53f3607_9f4c_47f6_8ddf_d81d67b0e38e.slice/crio-32caf39575af0cfb4eb48c5886d7642facce306b18f83e3be0b617d0f50e6470 WatchSource:0}: Error finding container 32caf39575af0cfb4eb48c5886d7642facce306b18f83e3be0b617d0f50e6470: Status 404 returned error can't find the container with id 32caf39575af0cfb4eb48c5886d7642facce306b18f83e3be0b617d0f50e6470 Jul 20 22:54:40.468831 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.468809 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86f6bc7688-gdrps"] Jul 20 22:54:40.471330 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:40.471306 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1395ccea_0612_4422_8bcc_050d534f6697.slice/crio-ab7a1525e569b363afe4f5962ae4ae096b4ed5e16d9fb73fa22773433897b429 WatchSource:0}: Error finding container ab7a1525e569b363afe4f5962ae4ae096b4ed5e16d9fb73fa22773433897b429: Status 404 returned error can't find the container with id ab7a1525e569b363afe4f5962ae4ae096b4ed5e16d9fb73fa22773433897b429 Jul 20 22:54:40.651178 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.651083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86f6bc7688-gdrps" event={"ID":"1395ccea-0612-4422-8bcc-050d534f6697","Type":"ContainerStarted","Data":"5c8ea0d1ea0f79a03a18ae2e1127db5710947253f817afd4f7dc0833cac01554"} Jul 20 22:54:40.651178 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.651144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86f6bc7688-gdrps" event={"ID":"1395ccea-0612-4422-8bcc-050d534f6697","Type":"ContainerStarted","Data":"ab7a1525e569b363afe4f5962ae4ae096b4ed5e16d9fb73fa22773433897b429"} Jul 20 22:54:40.652336 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.652314 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" event={"ID":"47303d5f-832f-48f6-98ea-3aa738f34b38","Type":"ContainerStarted","Data":"137dfee8c361bd9a163adb6c70b2cd44bc61bca1d6d6652f5736226df28131eb"} Jul 20 22:54:40.653495 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.653449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mm8pc" event={"ID":"e5af0766-d27e-4caf-bb82-e6929c31b8f9","Type":"ContainerStarted","Data":"979bfe64e76c0f91bc6b2adc459682f10d6c2b939753da9d63aa336fc85f0b84"} Jul 20 22:54:40.655049 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.655021 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" event={"ID":"359ef3fc-983f-4e37-982c-76b9ffb1adb5","Type":"ContainerStarted","Data":"132990df6f92561826df60963a6d91d24100244f4ff2b3132b40531dee496b22"} Jul 20 22:54:40.655165 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.655070 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" event={"ID":"359ef3fc-983f-4e37-982c-76b9ffb1adb5","Type":"ContainerStarted","Data":"894d8772e8a01cde18aeb8f38be644e8185117c02c351a5025c550b86aa7bf71"} Jul 20 22:54:40.655165 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.655099 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:54:40.656452 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.656423 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" event={"ID":"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e","Type":"ContainerStarted","Data":"32caf39575af0cfb4eb48c5886d7642facce306b18f83e3be0b617d0f50e6470"} Jul 20 22:54:40.657565 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.657546 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" event={"ID":"1caef281-adf8-4780-a381-47e481388395","Type":"ContainerStarted","Data":"e0251ec785bf1be00e915e4bd4a0e43ccb47757133dceac3da659679b24aff19"} Jul 20 22:54:40.658878 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.658853 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rhwct" event={"ID":"7febacb1-bee8-4309-9751-6e2d693f2bcc","Type":"ContainerStarted","Data":"1da20f52266a4b0690b763551eee2b15fc46ba91a1153bde064153f9a024ba15"} Jul 20 22:54:40.658986 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.658891 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:40.665655 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.665637 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:40.672992 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.672939 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-86f6bc7688-gdrps" podStartSLOduration=1.672919024 podStartE2EDuration="1.672919024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:40.671765042 +0000 UTC m=+55.990117329" watchObservedRunningTime="2026-07-20 22:54:40.672919024 +0000 UTC m=+55.991271290" Jul 20 22:54:40.712961 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.712911 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" podStartSLOduration=1.71289638 podStartE2EDuration="1.71289638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:40.712075173 +0000 UTC m=+56.030427459" watchObservedRunningTime="2026-07-20 22:54:40.71289638 +0000 UTC m=+56.031248643" Jul 20 22:54:40.838987 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.838959 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-trusted-ca\") pod \"dcbf07b6-9d17-4014-a290-0da81107815e\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " Jul 20 22:54:40.839149 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.839015 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-registry-tls\") pod \"dcbf07b6-9d17-4014-a290-0da81107815e\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " Jul 20 22:54:40.839149 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.839039 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-bound-sa-token\") pod \"dcbf07b6-9d17-4014-a290-0da81107815e\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " Jul 20 22:54:40.839149 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.839072 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-registry-certificates\") pod \"dcbf07b6-9d17-4014-a290-0da81107815e\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " Jul 20 22:54:40.839149 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.839104 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-installation-pull-secrets\") pod \"dcbf07b6-9d17-4014-a290-0da81107815e\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " Jul 20 22:54:40.839149 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.839135 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-image-registry-private-configuration\") pod \"dcbf07b6-9d17-4014-a290-0da81107815e\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " Jul 20 22:54:40.839418 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.839173 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvk67\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-kube-api-access-wvk67\") pod \"dcbf07b6-9d17-4014-a290-0da81107815e\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " Jul 20 22:54:40.839418 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.839208 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dcbf07b6-9d17-4014-a290-0da81107815e-ca-trust-extracted\") pod \"dcbf07b6-9d17-4014-a290-0da81107815e\" (UID: \"dcbf07b6-9d17-4014-a290-0da81107815e\") " Jul 20 22:54:40.839704 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.839651 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dcbf07b6-9d17-4014-a290-0da81107815e" (UID: "dcbf07b6-9d17-4014-a290-0da81107815e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 22:54:40.840831 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.840779 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbf07b6-9d17-4014-a290-0da81107815e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dcbf07b6-9d17-4014-a290-0da81107815e" (UID: "dcbf07b6-9d17-4014-a290-0da81107815e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 22:54:40.841003 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.840863 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dcbf07b6-9d17-4014-a290-0da81107815e" (UID: "dcbf07b6-9d17-4014-a290-0da81107815e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 22:54:40.842082 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.842032 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dcbf07b6-9d17-4014-a290-0da81107815e" (UID: "dcbf07b6-9d17-4014-a290-0da81107815e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:54:40.842279 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.842251 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dcbf07b6-9d17-4014-a290-0da81107815e" (UID: "dcbf07b6-9d17-4014-a290-0da81107815e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:54:40.842680 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.842654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dcbf07b6-9d17-4014-a290-0da81107815e" (UID: "dcbf07b6-9d17-4014-a290-0da81107815e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 22:54:40.842874 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.842841 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-kube-api-access-wvk67" (OuterVolumeSpecName: "kube-api-access-wvk67") pod "dcbf07b6-9d17-4014-a290-0da81107815e" (UID: "dcbf07b6-9d17-4014-a290-0da81107815e"). InnerVolumeSpecName "kube-api-access-wvk67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:54:40.842874 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.842847 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dcbf07b6-9d17-4014-a290-0da81107815e" (UID: "dcbf07b6-9d17-4014-a290-0da81107815e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 22:54:40.940650 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.940573 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-trusted-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:54:40.940650 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.940604 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-registry-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:54:40.940650 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.940615 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-bound-sa-token\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:54:40.940650 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.940627 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dcbf07b6-9d17-4014-a290-0da81107815e-registry-certificates\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:54:40.940650 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.940643 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-installation-pull-secrets\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:54:40.940650 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.940656 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dcbf07b6-9d17-4014-a290-0da81107815e-image-registry-private-configuration\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:54:40.940959 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.940670 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvk67\" (UniqueName: \"kubernetes.io/projected/dcbf07b6-9d17-4014-a290-0da81107815e-kube-api-access-wvk67\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:54:40.940959 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:40.940680 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dcbf07b6-9d17-4014-a290-0da81107815e-ca-trust-extracted\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:54:41.006202 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:41.006158 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:41.008840 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:41.008811 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:41.664792 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:41.664726 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:41.664792 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:41.664781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655fdccb8d-bcs6g" Jul 20 22:54:41.667297 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:41.667201 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-86f6bc7688-gdrps" Jul 20 22:54:41.699995 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:41.699932 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-655fdccb8d-bcs6g"] Jul 20 22:54:41.704018 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:41.703891 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-655fdccb8d-bcs6g"] Jul 20 22:54:43.230250 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:43.230050 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcbf07b6-9d17-4014-a290-0da81107815e" path="/var/lib/kubelet/pods/dcbf07b6-9d17-4014-a290-0da81107815e/volumes" Jul 20 22:54:45.599145 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:45.599113 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sjc24" Jul 20 22:54:47.107299 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.107262 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-clnd8"] Jul 20 22:54:47.110852 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.110820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.113680 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.113656 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 22:54:47.113817 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.113740 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 22:54:47.115355 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.115334 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 22:54:47.115481 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.115368 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 22:54:47.115481 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.115438 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lz6zc\"" Jul 20 22:54:47.123013 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.122988 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-clnd8"] Jul 20 22:54:47.194916 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.194882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07901246-5ff6-4ada-b81f-6b41ac1df630-data-volume\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.194916 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.194916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.195131 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.194946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.195131 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.194973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfk5x\" (UniqueName: \"kubernetes.io/projected/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-api-access-gfk5x\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.195230 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.195155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07901246-5ff6-4ada-b81f-6b41ac1df630-crio-socket\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.296302 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.296252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.296302 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.296308 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfk5x\" (UniqueName: \"kubernetes.io/projected/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-api-access-gfk5x\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.296590 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.296419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07901246-5ff6-4ada-b81f-6b41ac1df630-crio-socket\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.296590 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:47.296431 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 22:54:47.296590 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.296451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07901246-5ff6-4ada-b81f-6b41ac1df630-data-volume\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.296590 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.296489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.296590 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:47.296522 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls podName:07901246-5ff6-4ada-b81f-6b41ac1df630 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:47.796505928 +0000 UTC m=+63.114858192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clnd8" (UID: "07901246-5ff6-4ada-b81f-6b41ac1df630") : secret "insights-runtime-extractor-tls" not found Jul 20 22:54:47.296590 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.296548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07901246-5ff6-4ada-b81f-6b41ac1df630-crio-socket\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.296590 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:47.296570 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 22:54:47.296926 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:47.296634 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm podName:07901246-5ff6-4ada-b81f-6b41ac1df630 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:47.796617938 +0000 UTC m=+63.114970222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clnd8" (UID: "07901246-5ff6-4ada-b81f-6b41ac1df630") : configmap "kube-rbac-proxy" not found Jul 20 22:54:47.296926 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.296841 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07901246-5ff6-4ada-b81f-6b41ac1df630-data-volume\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.306904 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.306874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfk5x\" (UniqueName: \"kubernetes.io/projected/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-api-access-gfk5x\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.799829 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.799766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.800844 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:47.799997 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 22:54:47.800844 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:47.800094 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm podName:07901246-5ff6-4ada-b81f-6b41ac1df630 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:48.800071046 +0000 UTC m=+64.118423324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clnd8" (UID: "07901246-5ff6-4ada-b81f-6b41ac1df630") : configmap "kube-rbac-proxy" not found Jul 20 22:54:47.801092 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:47.800895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:47.801092 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:47.801044 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 22:54:47.801092 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:47.801082 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls podName:07901246-5ff6-4ada-b81f-6b41ac1df630 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:48.801071338 +0000 UTC m=+64.119423601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clnd8" (UID: "07901246-5ff6-4ada-b81f-6b41ac1df630") : secret "insights-runtime-extractor-tls" not found Jul 20 22:54:48.687880 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.687837 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" event={"ID":"47303d5f-832f-48f6-98ea-3aa738f34b38","Type":"ContainerStarted","Data":"713a6c87f4f3e8342fdede7b0e89799f8ecf9c485146e848e83d49ab5bd57dca"} Jul 20 22:54:48.689689 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.689658 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mm8pc" event={"ID":"e5af0766-d27e-4caf-bb82-e6929c31b8f9","Type":"ContainerStarted","Data":"9738fc76b2bf524af2752560d3f2fca82edb28a2bd85ee58f6cf9045829b997b"} Jul 20 22:54:48.689836 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.689712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mm8pc" event={"ID":"e5af0766-d27e-4caf-bb82-e6929c31b8f9","Type":"ContainerStarted","Data":"4c310db0c8f455d2226b6707fa7756a2908ef886cee24c9671f4e4a6dc880d21"} Jul 20 22:54:48.689836 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.689763 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:48.691129 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.691102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" event={"ID":"f53f3607-9f4c-47f6-8ddf-d81d67b0e38e","Type":"ContainerStarted","Data":"c918ab6400cd3abd7a59561d5fa76ce875b0a78d5c803d640d047fa63b2bf90a"} Jul 20 22:54:48.691292 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.691264 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:48.692859 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.692839 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" Jul 20 22:54:48.693412 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.693390 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" event={"ID":"1caef281-adf8-4780-a381-47e481388395","Type":"ContainerStarted","Data":"49bd3bc78b71d3470074f69e922b77cbbb01c82c08cd737353da0b28533d9301"} Jul 20 22:54:48.693538 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.693427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" event={"ID":"1caef281-adf8-4780-a381-47e481388395","Type":"ContainerStarted","Data":"645de02c9a016bb7666b092608be4347ff7c396589a06656a0b34cd0f1824169"} Jul 20 22:54:48.693538 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.693450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" event={"ID":"1caef281-adf8-4780-a381-47e481388395","Type":"ContainerStarted","Data":"ba41d2ef1490eeeaae86b6d8af165af3cbbbfcbf04c0e864e36f1c17513e4543"} Jul 20 22:54:48.694812 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.694788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rhwct" event={"ID":"7febacb1-bee8-4309-9751-6e2d693f2bcc","Type":"ContainerStarted","Data":"ec3b7e95eb106526bb857989b9d156f315b48f4b7ec500f528fe29ff8b3b6ced"} Jul 20 22:54:48.701861 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.701840 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mm8pc_e5af0766-d27e-4caf-bb82-e6929c31b8f9/kube-rbac-proxy/0.log" Jul 20 22:54:48.705957 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.705918 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-6d87c" podStartSLOduration=2.4341349 podStartE2EDuration="9.705906512s" totalImagesPullingTime="7.271771612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:48.704303879 +0000 UTC m=+64.022656164" watchObservedRunningTime="2026-07-20 22:54:48.705906512 +0000 UTC m=+64.024258798" Jul 20 22:54:48.721872 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.721825 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mm8pc" podStartSLOduration=2.461614883 podStartE2EDuration="9.721809051s" totalImagesPullingTime="7.260194168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:48.721468769 +0000 UTC m=+64.039821071" watchObservedRunningTime="2026-07-20 22:54:48.721809051 +0000 UTC m=+64.040161337" Jul 20 22:54:48.742631 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.742570 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc6b765d7-hqfcl" podStartSLOduration=2.452194956 podStartE2EDuration="9.742551051s" totalImagesPullingTime="7.290356095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:48.740720816 +0000 UTC m=+64.059073105" watchObservedRunningTime="2026-07-20 22:54:48.742551051 +0000 UTC m=+64.060903320" Jul 20 22:54:48.763174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.763124 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rhwct" podStartSLOduration=2.456175148 podStartE2EDuration="9.763107659s" totalImagesPullingTime="7.306932511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:48.762266219 +0000 UTC m=+64.080618506" watchObservedRunningTime="2026-07-20 22:54:48.763107659 +0000 UTC m=+64.081459945" Jul 20 22:54:48.782722 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.782674 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67f85965b9-sqglp" podStartSLOduration=2.4751547179999998 podStartE2EDuration="9.782656674s" totalImagesPullingTime="7.307501956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:48.781908574 +0000 UTC m=+64.100260861" watchObservedRunningTime="2026-07-20 22:54:48.782656674 +0000 UTC m=+64.101008959" Jul 20 22:54:48.809337 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.809307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:48.809337 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:48.809341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:48.809555 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:48.809423 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 22:54:48.809555 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:48.809435 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 22:54:48.809555 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:48.809517 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls podName:07901246-5ff6-4ada-b81f-6b41ac1df630 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:50.809504463 +0000 UTC m=+66.127856726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clnd8" (UID: "07901246-5ff6-4ada-b81f-6b41ac1df630") : secret "insights-runtime-extractor-tls" not found Jul 20 22:54:48.809555 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:48.809530 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm podName:07901246-5ff6-4ada-b81f-6b41ac1df630 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:50.809524092 +0000 UTC m=+66.127876356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clnd8" (UID: "07901246-5ff6-4ada-b81f-6b41ac1df630") : configmap "kube-rbac-proxy" not found Jul 20 22:54:49.100417 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.100388 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jx6hc_e84fa919-76a7-476e-9657-c195f0217bbc/dns-node-resolver/0.log" Jul 20 22:54:49.501143 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.501056 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84df66f6d-lzwgx_359ef3fc-983f-4e37-982c-76b9ffb1adb5/registry/0.log" Jul 20 22:54:49.818363 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.818270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:49.818750 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.818367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:49.821002 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.820986 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 22:54:49.821075 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.821041 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 22:54:49.831445 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.831421 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f29150c-6bc0-49cc-af95-18545831cf49-metrics-certs\") pod \"network-metrics-daemon-wpvj7\" (UID: \"7f29150c-6bc0-49cc-af95-18545831cf49\") " pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:49.831567 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.831505 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5be5526a-0c29-42b3-91fb-6c6863746390-original-pull-secret\") pod \"global-pull-secret-syncer-574vd\" (UID: \"5be5526a-0c29-42b3-91fb-6c6863746390\") " pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:49.919389 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.919353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:49.922174 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.922155 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 22:54:49.932537 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.932513 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 22:54:49.934979 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.934964 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d688j\"" Jul 20 22:54:49.942321 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.942300 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mxf\" (UniqueName: \"kubernetes.io/projected/a489a534-0ee3-4d50-9bd3-d77da326f316-kube-api-access-k7mxf\") pod \"network-check-target-ff7jr\" (UID: \"a489a534-0ee3-4d50-9bd3-d77da326f316\") " pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:49.942963 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.942943 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpvj7" Jul 20 22:54:49.946191 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:49.946172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-574vd" Jul 20 22:54:50.081220 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.081195 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wpvj7"] Jul 20 22:54:50.083592 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:50.083567 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f29150c_6bc0_49cc_af95_18545831cf49.slice/crio-9d43c7ead23710d1ffeedaf2cc635d335e2134106558781af10572f386ccb434 WatchSource:0}: Error finding container 9d43c7ead23710d1ffeedaf2cc635d335e2134106558781af10572f386ccb434: Status 404 returned error can't find the container with id 9d43c7ead23710d1ffeedaf2cc635d335e2134106558781af10572f386ccb434 Jul 20 22:54:50.096121 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.096080 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-574vd"] Jul 20 22:54:50.098522 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:50.098496 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be5526a_0c29_42b3_91fb_6c6863746390.slice/crio-d7c0ec8b710c24ea8a2eedf105c37be959755726fbdc7c75f7f5591e815bc5fc WatchSource:0}: Error finding container d7c0ec8b710c24ea8a2eedf105c37be959755726fbdc7c75f7f5591e815bc5fc: Status 404 returned error can't find the container with id d7c0ec8b710c24ea8a2eedf105c37be959755726fbdc7c75f7f5591e815bc5fc Jul 20 22:54:50.100430 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.100406 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cmrt4_0474b40b-5711-4a08-a644-f9caf4afed37/node-ca/0.log" Jul 20 22:54:50.243639 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.243606 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xr56f\"" Jul 20 22:54:50.251107 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.251083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:50.380666 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.380519 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ff7jr"] Jul 20 22:54:50.383061 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:50.383032 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda489a534_0ee3_4d50_9bd3_d77da326f316.slice/crio-f4ff3332471730e2e85125355fd407fb5486f9aaa67505d0b2fbf64ff23e07c7 WatchSource:0}: Error finding container f4ff3332471730e2e85125355fd407fb5486f9aaa67505d0b2fbf64ff23e07c7: Status 404 returned error can't find the container with id f4ff3332471730e2e85125355fd407fb5486f9aaa67505d0b2fbf64ff23e07c7 Jul 20 22:54:50.501397 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.501369 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86f6bc7688-gdrps_1395ccea-0612-4422-8bcc-050d534f6697/router/0.log" Jul 20 22:54:50.704451 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.704409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-574vd" event={"ID":"5be5526a-0c29-42b3-91fb-6c6863746390","Type":"ContainerStarted","Data":"d7c0ec8b710c24ea8a2eedf105c37be959755726fbdc7c75f7f5591e815bc5fc"} Jul 20 22:54:50.706046 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.706016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wpvj7" event={"ID":"7f29150c-6bc0-49cc-af95-18545831cf49","Type":"ContainerStarted","Data":"9d43c7ead23710d1ffeedaf2cc635d335e2134106558781af10572f386ccb434"} Jul 20 22:54:50.707399 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.707345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ff7jr" event={"ID":"a489a534-0ee3-4d50-9bd3-d77da326f316","Type":"ContainerStarted","Data":"f4ff3332471730e2e85125355fd407fb5486f9aaa67505d0b2fbf64ff23e07c7"} Jul 20 22:54:50.825918 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.825887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:50.825918 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:50.825929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:50.826417 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:50.826033 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 22:54:50.826417 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:50.826073 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 22:54:50.826417 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:50.826136 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm podName:07901246-5ff6-4ada-b81f-6b41ac1df630 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:54.826114777 +0000 UTC m=+70.144467052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clnd8" (UID: "07901246-5ff6-4ada-b81f-6b41ac1df630") : configmap "kube-rbac-proxy" not found Jul 20 22:54:50.826417 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:50.826157 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls podName:07901246-5ff6-4ada-b81f-6b41ac1df630 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:54.826147139 +0000 UTC m=+70.144499410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clnd8" (UID: "07901246-5ff6-4ada-b81f-6b41ac1df630") : secret "insights-runtime-extractor-tls" not found Jul 20 22:54:51.103751 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:51.103719 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rhwct_7febacb1-bee8-4309-9751-6e2d693f2bcc/serve-healthcheck-canary/0.log" Jul 20 22:54:51.714019 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:51.713978 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wpvj7" event={"ID":"7f29150c-6bc0-49cc-af95-18545831cf49","Type":"ContainerStarted","Data":"a5423189715d43f4d16fe8a4de744c7a78b9a543dd72747bb277c215db9fcbb2"} Jul 20 22:54:51.714187 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:51.714047 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wpvj7" event={"ID":"7f29150c-6bc0-49cc-af95-18545831cf49","Type":"ContainerStarted","Data":"2fbdc83a813ee13966e7a4ca38b06c5ba997a5d21126f918f2661a8f9ca0a008"} Jul 20 22:54:51.732267 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:51.732203 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wpvj7" podStartSLOduration=65.701702068 podStartE2EDuration="1m6.73217985s" totalImagesPullingTime="1.030477782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:51.731710213 +0000 UTC m=+67.050062502" watchObservedRunningTime="2026-07-20 22:54:51.73217985 +0000 UTC m=+67.050532136" Jul 20 22:54:54.861168 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:54.861129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:54.861168 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:54.861175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:54.861845 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:54.861812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07901246-5ff6-4ada-b81f-6b41ac1df630-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:54.863909 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:54.863883 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07901246-5ff6-4ada-b81f-6b41ac1df630-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clnd8\" (UID: \"07901246-5ff6-4ada-b81f-6b41ac1df630\") " pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:54.921006 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:54.920973 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-clnd8" Jul 20 22:54:55.278796 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:55.278752 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-clnd8"] Jul 20 22:54:55.282942 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:54:55.282912 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07901246_5ff6_4ada_b81f_6b41ac1df630.slice/crio-6a15c012ab301aec73cb6fcebb9493a08353f0ec5a9daaf70997ae71a80bfb9f WatchSource:0}: Error finding container 6a15c012ab301aec73cb6fcebb9493a08353f0ec5a9daaf70997ae71a80bfb9f: Status 404 returned error can't find the container with id 6a15c012ab301aec73cb6fcebb9493a08353f0ec5a9daaf70997ae71a80bfb9f Jul 20 22:54:55.733726 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:55.733427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ff7jr" event={"ID":"a489a534-0ee3-4d50-9bd3-d77da326f316","Type":"ContainerStarted","Data":"7d528c576781c3fe6c75ee0446e8c4149bc05d20c9184938f7b51807a1b2627c"} Jul 20 22:54:55.733726 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:55.733537 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:54:55.736517 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:55.736488 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-574vd" event={"ID":"5be5526a-0c29-42b3-91fb-6c6863746390","Type":"ContainerStarted","Data":"3ec152636766ae6cd4759661ae947fe518ea43115fcb5afe7f6c5da9ae2d233f"} Jul 20 22:54:55.739133 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:55.739069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clnd8" event={"ID":"07901246-5ff6-4ada-b81f-6b41ac1df630","Type":"ContainerStarted","Data":"6a15c012ab301aec73cb6fcebb9493a08353f0ec5a9daaf70997ae71a80bfb9f"} Jul 20 22:54:55.753088 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:55.753037 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ff7jr" podStartSLOduration=65.994725363 podStartE2EDuration="1m10.753020123s" totalImagesPullingTime="4.75829476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:55.751727085 +0000 UTC m=+71.070079373" watchObservedRunningTime="2026-07-20 22:54:55.753020123 +0000 UTC m=+71.071372405" Jul 20 22:54:55.769265 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:55.769032 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-574vd" podStartSLOduration=65.729757882 podStartE2EDuration="1m10.769009857s" totalImagesPullingTime="5.039251975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:55.767626143 +0000 UTC m=+71.085978444" watchObservedRunningTime="2026-07-20 22:54:55.769009857 +0000 UTC m=+71.087362144" Jul 20 22:54:58.678159 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.678120 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zshqq"] Jul 20 22:54:58.704134 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.704097 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mm8pc" Jul 20 22:54:58.704298 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.704190 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.706988 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.706961 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 22:54:58.707294 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.706979 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 22:54:58.707384 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.707330 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xfqr8\"" Jul 20 22:54:58.708799 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.708774 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 22:54:58.708799 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.708803 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 22:54:58.794708 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.794672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-metrics-client-ca\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.794869 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.794729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-root\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.794869 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.794800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-tls\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.794869 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.794837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-accelerators-collector-config\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.794869 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.794862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjng7\" (UniqueName: \"kubernetes.io/projected/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-kube-api-access-fjng7\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.795034 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.794910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-sys\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.795034 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.794943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-wtmp\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.795034 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.794966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-textfile\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.795034 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.795012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896059 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-root\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896059 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-tls\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896306 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-accelerators-collector-config\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896306 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjng7\" (UniqueName: \"kubernetes.io/projected/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-kube-api-access-fjng7\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896306 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-sys\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896306 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-root\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896306 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:58.896210 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 20 22:54:58.896306 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:54:58.896275 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-tls podName:f87265f1-e5ae-4d8e-ad1a-b82996c5578f nodeName:}" failed. No retries permitted until 2026-07-20 22:54:59.396255371 +0000 UTC m=+74.714607642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-tls") pod "node-exporter-zshqq" (UID: "f87265f1-e5ae-4d8e-ad1a-b82996c5578f") : secret "node-exporter-tls" not found Jul 20 22:54:58.896306 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-wtmp\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896685 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896342 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-sys\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896685 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896185 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-wtmp\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896685 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-textfile\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896685 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896685 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-metrics-client-ca\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896685 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-textfile\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896946 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-accelerators-collector-config\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.896946 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.896926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-metrics-client-ca\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.898920 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.898896 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:58.905500 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:58.905476 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjng7\" (UniqueName: \"kubernetes.io/projected/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-kube-api-access-fjng7\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:59.401406 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:59.401370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-tls\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:59.403611 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:59.403587 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f87265f1-e5ae-4d8e-ad1a-b82996c5578f-node-exporter-tls\") pod \"node-exporter-zshqq\" (UID: \"f87265f1-e5ae-4d8e-ad1a-b82996c5578f\") " pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:54:59.614857 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:54:59.614812 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zshqq" Jul 20 22:55:00.042111 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:55:00.042081 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87265f1_e5ae_4d8e_ad1a_b82996c5578f.slice/crio-bb50a7fc712c4dc07748ca386126cc2384482a7358b882263c56a6373a943c43 WatchSource:0}: Error finding container bb50a7fc712c4dc07748ca386126cc2384482a7358b882263c56a6373a943c43: Status 404 returned error can't find the container with id bb50a7fc712c4dc07748ca386126cc2384482a7358b882263c56a6373a943c43 Jul 20 22:55:00.171639 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:00.171601 2574 patch_prober.go:28] interesting pod/image-registry-84df66f6d-lzwgx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 20 22:55:00.171780 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:00.171667 2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" podUID="359ef3fc-983f-4e37-982c-76b9ffb1adb5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 22:55:00.761694 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:00.761652 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clnd8" event={"ID":"07901246-5ff6-4ada-b81f-6b41ac1df630","Type":"ContainerStarted","Data":"237294348e1271d309563a03bac62b0502942c950316330350ab6ead18bbb92b"} Jul 20 22:55:00.763520 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:00.763374 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zshqq" event={"ID":"f87265f1-e5ae-4d8e-ad1a-b82996c5578f","Type":"ContainerStarted","Data":"bb50a7fc712c4dc07748ca386126cc2384482a7358b882263c56a6373a943c43"} Jul 20 22:55:01.670403 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:01.670368 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84df66f6d-lzwgx" Jul 20 22:55:01.771783 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:01.771746 2574 generic.go:356] "Generic (PLEG): container finished" podID="f87265f1-e5ae-4d8e-ad1a-b82996c5578f" containerID="6ac3fd4200c4725414481619a16a97e507bb9b078a85ba999e2f3d64fbf0debc" exitCode=0 Jul 20 22:55:01.771955 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:01.771826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zshqq" event={"ID":"f87265f1-e5ae-4d8e-ad1a-b82996c5578f","Type":"ContainerDied","Data":"6ac3fd4200c4725414481619a16a97e507bb9b078a85ba999e2f3d64fbf0debc"} Jul 20 22:55:03.780980 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:03.780939 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clnd8" event={"ID":"07901246-5ff6-4ada-b81f-6b41ac1df630","Type":"ContainerStarted","Data":"64bf24a87f478412af31ec6ba4afd18060bc95929f3bc4a2b2661e09c341c3c3"} Jul 20 22:55:03.782927 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:03.782897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zshqq" event={"ID":"f87265f1-e5ae-4d8e-ad1a-b82996c5578f","Type":"ContainerStarted","Data":"5a538f7bf31e504993eb0276784b3695f6f237929435b40ddcc15ee0f4bb900a"} Jul 20 22:55:03.783072 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:03.782948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zshqq" event={"ID":"f87265f1-e5ae-4d8e-ad1a-b82996c5578f","Type":"ContainerStarted","Data":"2a7d7d0bf074e1f314654e43a4ded775172de555f0e1d7c65849d673b0559446"} Jul 20 22:55:03.807330 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:03.807224 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zshqq" podStartSLOduration=4.833374696 podStartE2EDuration="5.807209151s" totalImagesPullingTime="898.881512ms" totalInitContainerRuntime="74.952943ms" isStatefulPod=true podCreationTimestamp="2026-07-20 22:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:55:03.805519376 +0000 UTC m=+79.123871663" watchObservedRunningTime="2026-07-20 22:55:03.807209151 +0000 UTC m=+79.125561436" Jul 20 22:55:07.800990 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:07.800955 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clnd8" event={"ID":"07901246-5ff6-4ada-b81f-6b41ac1df630","Type":"ContainerStarted","Data":"c59f3fa65b0f162c5e2824cdc0c2f9666a93cf5ad9e3537497d8b5dac6ec48d2"} Jul 20 22:55:07.824134 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:07.824081 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-clnd8" podStartSLOduration=9.035296258 podStartE2EDuration="20.824065001s" totalImagesPullingTime="11.788768743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:54:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:55:07.822289846 +0000 UTC m=+83.140642143" watchObservedRunningTime="2026-07-20 22:55:07.824065001 +0000 UTC m=+83.142417295" Jul 20 22:55:26.745785 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:55:26.745753 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ff7jr" Jul 20 22:58:45.031963 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:45.031931 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 22:58:45.038706 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:45.038680 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 22:58:45.040509 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:45.040487 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 22:58:54.052077 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.052018 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp"] Jul 20 22:58:54.055278 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.055261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.059267 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.059244 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 22:58:54.059452 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.059279 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 22:58:54.061206 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.060943 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 22:58:54.061206 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.060943 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nptdq\"" Jul 20 22:58:54.061206 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.060943 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 22:58:54.097438 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.097406 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp"] Jul 20 22:58:54.178699 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.178662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvbk\" (UniqueName: \"kubernetes.io/projected/b91ee607-5707-4410-9e32-ee61b9ea3204-kube-api-access-gxvbk\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.178699 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.178703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b91ee607-5707-4410-9e32-ee61b9ea3204-tmp\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.178900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.178761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b91ee607-5707-4410-9e32-ee61b9ea3204-webhook-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.178900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.178808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b91ee607-5707-4410-9e32-ee61b9ea3204-manifests\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.178900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.178849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b91ee607-5707-4410-9e32-ee61b9ea3204-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.279379 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.279331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b91ee607-5707-4410-9e32-ee61b9ea3204-webhook-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.279379 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.279386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b91ee607-5707-4410-9e32-ee61b9ea3204-manifests\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.279597 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.279499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b91ee607-5707-4410-9e32-ee61b9ea3204-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.279597 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.279587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvbk\" (UniqueName: \"kubernetes.io/projected/b91ee607-5707-4410-9e32-ee61b9ea3204-kube-api-access-gxvbk\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.279667 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.279611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b91ee607-5707-4410-9e32-ee61b9ea3204-tmp\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.279851 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.279829 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b91ee607-5707-4410-9e32-ee61b9ea3204-manifests\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.281674 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.281655 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b91ee607-5707-4410-9e32-ee61b9ea3204-tmp\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.282020 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.281999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b91ee607-5707-4410-9e32-ee61b9ea3204-webhook-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.282020 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.282010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b91ee607-5707-4410-9e32-ee61b9ea3204-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.287891 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.287867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvbk\" (UniqueName: \"kubernetes.io/projected/b91ee607-5707-4410-9e32-ee61b9ea3204-kube-api-access-gxvbk\") pod \"opendatahub-operator-controller-manager-699fd79659-kxtvp\" (UID: \"b91ee607-5707-4410-9e32-ee61b9ea3204\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.364929 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.364896 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:58:54.500727 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.500700 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp"] Jul 20 22:58:54.504047 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:58:54.504016 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91ee607_5707_4410_9e32_ee61b9ea3204.slice/crio-22c814a557e0b0b9216e3739df99e00c9cca3bf12ee67c66c0fef3aab2310b07 WatchSource:0}: Error finding container 22c814a557e0b0b9216e3739df99e00c9cca3bf12ee67c66c0fef3aab2310b07: Status 404 returned error can't find the container with id 22c814a557e0b0b9216e3739df99e00c9cca3bf12ee67c66c0fef3aab2310b07 Jul 20 22:58:54.505840 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.505821 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 22:58:54.543566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:54.543529 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" event={"ID":"b91ee607-5707-4410-9e32-ee61b9ea3204","Type":"ContainerStarted","Data":"22c814a557e0b0b9216e3739df99e00c9cca3bf12ee67c66c0fef3aab2310b07"} Jul 20 22:58:58.559496 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:58.559429 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" event={"ID":"b91ee607-5707-4410-9e32-ee61b9ea3204","Type":"ContainerStarted","Data":"920c377fa8d84b91b1a16a8a43ef3e24924d93f1120ac86afb71314460294e63"} Jul 20 22:58:59.564504 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:59.564450 2574 generic.go:356] "Generic (PLEG): container finished" podID="b91ee607-5707-4410-9e32-ee61b9ea3204" containerID="920c377fa8d84b91b1a16a8a43ef3e24924d93f1120ac86afb71314460294e63" exitCode=0 Jul 20 22:58:59.564504 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:58:59.564493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" event={"ID":"b91ee607-5707-4410-9e32-ee61b9ea3204","Type":"ContainerDied","Data":"920c377fa8d84b91b1a16a8a43ef3e24924d93f1120ac86afb71314460294e63"} Jul 20 22:59:00.569566 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:00.569527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" event={"ID":"b91ee607-5707-4410-9e32-ee61b9ea3204","Type":"ContainerStarted","Data":"02cdc916bbcf123c4e93a5f90724a01b23be71e729233f159d185b9beabee761"} Jul 20 22:59:00.569938 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:00.569631 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:59:00.596178 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:00.596132 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" podStartSLOduration=2.146840023 podStartE2EDuration="6.596117503s" totalImagesPullingTime="3.413543269s" totalInitContainerRuntime="1.035734211s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:58:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:00.595957784 +0000 UTC m=+315.914310070" watchObservedRunningTime="2026-07-20 22:59:00.596117503 +0000 UTC m=+315.914469788" Jul 20 22:59:11.575099 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:11.575025 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-kxtvp" Jul 20 22:59:13.724160 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.724108 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5"] Jul 20 22:59:13.731626 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.731603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:13.734691 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.734669 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7mfrk\"" Jul 20 22:59:13.735526 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.735501 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 20 22:59:13.735845 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.735831 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 20 22:59:13.739741 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.739713 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5"] Jul 20 22:59:13.836133 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.836095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct4dg\" (UniqueName: \"kubernetes.io/projected/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-kube-api-access-ct4dg\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:13.836307 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.836154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-tls-certs\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:13.836307 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.836220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-tmp\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:13.936877 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.936817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-tls-certs\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:13.937060 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.936977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-tmp\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:13.937060 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.937040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct4dg\" (UniqueName: \"kubernetes.io/projected/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-kube-api-access-ct4dg\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:13.937178 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:59:13.936979 2574 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 20 22:59:13.937234 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:59:13.937196 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-tls-certs podName:8dc6d214-d9c2-4e4e-9b75-51a41b9c303c nodeName:}" failed. No retries permitted until 2026-07-20 22:59:14.437169682 +0000 UTC m=+329.755521960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-tls-certs") pod "kube-auth-proxy-85d6bc8fc7-6m8t5" (UID: "8dc6d214-d9c2-4e4e-9b75-51a41b9c303c") : secret "kube-auth-proxy-tls" not found Jul 20 22:59:13.939110 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.939086 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-tmp\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:13.966659 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:13.966629 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct4dg\" (UniqueName: \"kubernetes.io/projected/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-kube-api-access-ct4dg\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:14.442490 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.442424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-tls-certs\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:14.444910 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.444883 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc6d214-d9c2-4e4e-9b75-51a41b9c303c-tls-certs\") pod \"kube-auth-proxy-85d6bc8fc7-6m8t5\" (UID: \"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:14.640922 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.640885 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" Jul 20 22:59:14.773332 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.773265 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5"] Jul 20 22:59:14.775496 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:59:14.775447 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc6d214_d9c2_4e4e_9b75_51a41b9c303c.slice/crio-d51ee61a53bfb9a1400faa93a3bcfa28283f216076cd0072941972833445ed31 WatchSource:0}: Error finding container d51ee61a53bfb9a1400faa93a3bcfa28283f216076cd0072941972833445ed31: Status 404 returned error can't find the container with id d51ee61a53bfb9a1400faa93a3bcfa28283f216076cd0072941972833445ed31 Jul 20 22:59:14.803986 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.803945 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tnwnd"] Jul 20 22:59:14.807994 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.807975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:14.810900 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.810875 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 20 22:59:14.819099 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.819067 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tnwnd"] Jul 20 22:59:14.947030 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.946995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9775d2c-3396-4de8-8f9a-58fb144581d0-tls-certs\") pod \"model-serving-api-54d89bd5fc-tnwnd\" (UID: \"d9775d2c-3396-4de8-8f9a-58fb144581d0\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:14.947183 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:14.947048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7schs\" (UniqueName: \"kubernetes.io/projected/d9775d2c-3396-4de8-8f9a-58fb144581d0-kube-api-access-7schs\") pod \"model-serving-api-54d89bd5fc-tnwnd\" (UID: \"d9775d2c-3396-4de8-8f9a-58fb144581d0\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:15.046341 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.046258 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm"] Jul 20 22:59:15.048139 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.048114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9775d2c-3396-4de8-8f9a-58fb144581d0-tls-certs\") pod \"model-serving-api-54d89bd5fc-tnwnd\" (UID: \"d9775d2c-3396-4de8-8f9a-58fb144581d0\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:15.048259 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.048167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7schs\" (UniqueName: \"kubernetes.io/projected/d9775d2c-3396-4de8-8f9a-58fb144581d0-kube-api-access-7schs\") pod \"model-serving-api-54d89bd5fc-tnwnd\" (UID: \"d9775d2c-3396-4de8-8f9a-58fb144581d0\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:15.048324 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:59:15.048270 2574 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 20 22:59:15.048373 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:59:15.048334 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9775d2c-3396-4de8-8f9a-58fb144581d0-tls-certs podName:d9775d2c-3396-4de8-8f9a-58fb144581d0 nodeName:}" failed. No retries permitted until 2026-07-20 22:59:15.548317457 +0000 UTC m=+330.866669726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d9775d2c-3396-4de8-8f9a-58fb144581d0-tls-certs") pod "model-serving-api-54d89bd5fc-tnwnd" (UID: "d9775d2c-3396-4de8-8f9a-58fb144581d0") : secret "model-serving-api-tls" not found Jul 20 22:59:15.049240 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.049222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.051973 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.051941 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 20 22:59:15.052076 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.051975 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 20 22:59:15.052076 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.052018 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 20 22:59:15.052189 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.052074 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6mrk5\"" Jul 20 22:59:15.052307 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.052292 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 22:59:15.060342 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.060320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7schs\" (UniqueName: \"kubernetes.io/projected/d9775d2c-3396-4de8-8f9a-58fb144581d0-kube-api-access-7schs\") pod \"model-serving-api-54d89bd5fc-tnwnd\" (UID: \"d9775d2c-3396-4de8-8f9a-58fb144581d0\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:15.063448 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.063426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm"] Jul 20 22:59:15.149477 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.149421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.149477 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.149480 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.149676 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.149538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.149676 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.149559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvg4t\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-kube-api-access-wvg4t\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.149676 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.149617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-cacerts\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.149676 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.149666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ad569-2394-4b70-a45e-c48fc4c8b00e-local-certs\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.149801 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.149688 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-token\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.250622 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.250587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.250622 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.250628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvg4t\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-kube-api-access-wvg4t\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.250855 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.250658 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-cacerts\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.250855 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.250685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ad569-2394-4b70-a45e-c48fc4c8b00e-local-certs\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.250855 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.250709 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-token\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.250855 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.250790 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.250855 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.250812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.251553 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.251524 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.253147 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.253129 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ad569-2394-4b70-a45e-c48fc4c8b00e-local-certs\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.253250 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.253173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.253450 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.253433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-cacerts\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.253578 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.253560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.259571 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.259552 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-token\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.259690 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.259670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvg4t\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-kube-api-access-wvg4t\") pod \"istiod-openshift-gateway-7574459656-6mrrm\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.358825 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.358790 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:15.511691 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.511665 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm"] Jul 20 22:59:15.514221 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:59:15.514194 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32ad569_2394_4b70_a45e_c48fc4c8b00e.slice/crio-ca26f6b6994ffc7f71629273f03366078cff69dd7a0d43bbe3ae0fc1ac6ace3b WatchSource:0}: Error finding container ca26f6b6994ffc7f71629273f03366078cff69dd7a0d43bbe3ae0fc1ac6ace3b: Status 404 returned error can't find the container with id ca26f6b6994ffc7f71629273f03366078cff69dd7a0d43bbe3ae0fc1ac6ace3b Jul 20 22:59:15.554629 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.554592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9775d2c-3396-4de8-8f9a-58fb144581d0-tls-certs\") pod \"model-serving-api-54d89bd5fc-tnwnd\" (UID: \"d9775d2c-3396-4de8-8f9a-58fb144581d0\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:15.557508 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.557476 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9775d2c-3396-4de8-8f9a-58fb144581d0-tls-certs\") pod \"model-serving-api-54d89bd5fc-tnwnd\" (UID: \"d9775d2c-3396-4de8-8f9a-58fb144581d0\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:15.627272 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.627173 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" event={"ID":"e32ad569-2394-4b70-a45e-c48fc4c8b00e","Type":"ContainerStarted","Data":"ca26f6b6994ffc7f71629273f03366078cff69dd7a0d43bbe3ae0fc1ac6ace3b"} Jul 20 22:59:15.628402 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.628372 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" event={"ID":"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c","Type":"ContainerStarted","Data":"d51ee61a53bfb9a1400faa93a3bcfa28283f216076cd0072941972833445ed31"} Jul 20 22:59:15.718064 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.718031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:15.865120 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:15.865021 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tnwnd"] Jul 20 22:59:15.867251 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:59:15.867217 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9775d2c_3396_4de8_8f9a_58fb144581d0.slice/crio-7c021444b1557c191fe2bec567ba0ab58e5e781b151932c90a4249c5ee12926a WatchSource:0}: Error finding container 7c021444b1557c191fe2bec567ba0ab58e5e781b151932c90a4249c5ee12926a: Status 404 returned error can't find the container with id 7c021444b1557c191fe2bec567ba0ab58e5e781b151932c90a4249c5ee12926a Jul 20 22:59:16.634690 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:16.634645 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" event={"ID":"d9775d2c-3396-4de8-8f9a-58fb144581d0","Type":"ContainerStarted","Data":"7c021444b1557c191fe2bec567ba0ab58e5e781b151932c90a4249c5ee12926a"} Jul 20 22:59:18.633977 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:18.633945 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 22:59:18.634339 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:18.634025 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 22:59:19.648153 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.648118 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" event={"ID":"8dc6d214-d9c2-4e4e-9b75-51a41b9c303c","Type":"ContainerStarted","Data":"ccc72b9aaefcb72e849953b3c652c8e5b1b0a8e6cdd538f4cf2c7056f1c50c40"} Jul 20 22:59:19.649661 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.649634 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" event={"ID":"d9775d2c-3396-4de8-8f9a-58fb144581d0","Type":"ContainerStarted","Data":"ee38a5444533bb43cab599db50de2f2efdd9978df50c3b2411900b758bb30428"} Jul 20 22:59:19.649735 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.649691 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:19.651077 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.651047 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" event={"ID":"e32ad569-2394-4b70-a45e-c48fc4c8b00e","Type":"ContainerStarted","Data":"8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20"} Jul 20 22:59:19.651270 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.651251 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:19.652892 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.652871 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-6mrrm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 22:59:19.652972 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.652912 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" podUID="e32ad569-2394-4b70-a45e-c48fc4c8b00e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 22:59:19.664768 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.664726 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-6m8t5" podStartSLOduration=2.811967337 podStartE2EDuration="6.664712735s" totalImagesPullingTime="3.852745398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:19.663592995 +0000 UTC m=+334.981945281" watchObservedRunningTime="2026-07-20 22:59:19.664712735 +0000 UTC m=+334.983065020" Jul 20 22:59:19.684324 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.684273 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" podStartSLOduration=2.919131475 podStartE2EDuration="5.684257109s" totalImagesPullingTime="2.765125634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:19.683245994 +0000 UTC m=+335.001598276" watchObservedRunningTime="2026-07-20 22:59:19.684257109 +0000 UTC m=+335.002609392" Jul 20 22:59:19.705333 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:19.705283 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" podStartSLOduration=1.58771332 podStartE2EDuration="4.705264563s" totalImagesPullingTime="3.117551243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:19.703391444 +0000 UTC m=+335.021743731" watchObservedRunningTime="2026-07-20 22:59:19.705264563 +0000 UTC m=+335.023616849" Jul 20 22:59:20.655303 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:20.655275 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:23.991724 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:23.991679 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7t68m"] Jul 20 22:59:23.997085 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:23.997058 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:23.999679 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:23.999654 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 20 22:59:23.999793 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:23.999725 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bcxrq\"" Jul 20 22:59:24.006072 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.006046 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7t68m"] Jul 20 22:59:24.007967 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.007945 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h"] Jul 20 22:59:24.010962 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.010945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:24.013674 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.013656 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 20 22:59:24.013810 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.013716 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-m9jk8\"" Jul 20 22:59:24.020921 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.020889 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h"] Jul 20 22:59:24.030102 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.030072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9zd\" (UniqueName: \"kubernetes.io/projected/d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2-kube-api-access-dn9zd\") pod \"llmisvc-controller-manager-777c6bd7d5-bn26h\" (UID: \"d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:24.030232 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.030127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf87b6dd-57f0-4a91-8750-f7599613751d-cert\") pod \"kserve-controller-manager-d6f4b65b8-7t68m\" (UID: \"cf87b6dd-57f0-4a91-8750-f7599613751d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:24.030232 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.030202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdh4s\" (UniqueName: \"kubernetes.io/projected/cf87b6dd-57f0-4a91-8750-f7599613751d-kube-api-access-kdh4s\") pod \"kserve-controller-manager-d6f4b65b8-7t68m\" (UID: \"cf87b6dd-57f0-4a91-8750-f7599613751d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:24.030342 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.030262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bn26h\" (UID: \"d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:24.131314 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.131282 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdh4s\" (UniqueName: \"kubernetes.io/projected/cf87b6dd-57f0-4a91-8750-f7599613751d-kube-api-access-kdh4s\") pod \"kserve-controller-manager-d6f4b65b8-7t68m\" (UID: \"cf87b6dd-57f0-4a91-8750-f7599613751d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:24.131601 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.131329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bn26h\" (UID: \"d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:24.131601 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.131407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9zd\" (UniqueName: \"kubernetes.io/projected/d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2-kube-api-access-dn9zd\") pod \"llmisvc-controller-manager-777c6bd7d5-bn26h\" (UID: \"d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:24.131601 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.131494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf87b6dd-57f0-4a91-8750-f7599613751d-cert\") pod \"kserve-controller-manager-d6f4b65b8-7t68m\" (UID: \"cf87b6dd-57f0-4a91-8750-f7599613751d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:24.131601 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:59:24.131541 2574 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 20 22:59:24.131601 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:59:24.131592 2574 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 20 22:59:24.131866 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:59:24.131616 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2-cert podName:d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2 nodeName:}" failed. No retries permitted until 2026-07-20 22:59:24.631595137 +0000 UTC m=+339.949947419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2-cert") pod "llmisvc-controller-manager-777c6bd7d5-bn26h" (UID: "d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2") : secret "llmisvc-webhook-server-cert" not found Jul 20 22:59:24.131866 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:59:24.131641 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf87b6dd-57f0-4a91-8750-f7599613751d-cert podName:cf87b6dd-57f0-4a91-8750-f7599613751d nodeName:}" failed. No retries permitted until 2026-07-20 22:59:24.631626164 +0000 UTC m=+339.949978429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cf87b6dd-57f0-4a91-8750-f7599613751d-cert") pod "kserve-controller-manager-d6f4b65b8-7t68m" (UID: "cf87b6dd-57f0-4a91-8750-f7599613751d") : secret "kserve-webhook-server-cert" not found Jul 20 22:59:24.142518 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.142484 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdh4s\" (UniqueName: \"kubernetes.io/projected/cf87b6dd-57f0-4a91-8750-f7599613751d-kube-api-access-kdh4s\") pod \"kserve-controller-manager-d6f4b65b8-7t68m\" (UID: \"cf87b6dd-57f0-4a91-8750-f7599613751d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:24.143116 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.143095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9zd\" (UniqueName: \"kubernetes.io/projected/d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2-kube-api-access-dn9zd\") pod \"llmisvc-controller-manager-777c6bd7d5-bn26h\" (UID: \"d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:24.636078 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.636038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf87b6dd-57f0-4a91-8750-f7599613751d-cert\") pod \"kserve-controller-manager-d6f4b65b8-7t68m\" (UID: \"cf87b6dd-57f0-4a91-8750-f7599613751d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:24.636266 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.636088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bn26h\" (UID: \"d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:24.638381 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.638354 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf87b6dd-57f0-4a91-8750-f7599613751d-cert\") pod \"kserve-controller-manager-d6f4b65b8-7t68m\" (UID: \"cf87b6dd-57f0-4a91-8750-f7599613751d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:24.638504 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.638432 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bn26h\" (UID: \"d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:24.909373 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.909284 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:24.922823 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:24.922795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:25.054990 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:25.054964 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7t68m"] Jul 20 22:59:25.057861 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:59:25.057833 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf87b6dd_57f0_4a91_8750_f7599613751d.slice/crio-47b364f4808097ea13e90af51e4f7ffa32e09be18a267220f069b5b755257496 WatchSource:0}: Error finding container 47b364f4808097ea13e90af51e4f7ffa32e09be18a267220f069b5b755257496: Status 404 returned error can't find the container with id 47b364f4808097ea13e90af51e4f7ffa32e09be18a267220f069b5b755257496 Jul 20 22:59:25.078306 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:25.078278 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h"] Jul 20 22:59:25.080588 ip-10-0-129-209 kubenswrapper[2574]: W0720 22:59:25.080562 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4474cbe_0c59_4d8e_8cc5_e90b3eda8ba2.slice/crio-ec259887df5eeb4346c2bcdd55ba00d01fcfe4914a4309d2054a43d2ea093331 WatchSource:0}: Error finding container ec259887df5eeb4346c2bcdd55ba00d01fcfe4914a4309d2054a43d2ea093331: Status 404 returned error can't find the container with id ec259887df5eeb4346c2bcdd55ba00d01fcfe4914a4309d2054a43d2ea093331 Jul 20 22:59:25.672995 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:25.672956 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" event={"ID":"cf87b6dd-57f0-4a91-8750-f7599613751d","Type":"ContainerStarted","Data":"47b364f4808097ea13e90af51e4f7ffa32e09be18a267220f069b5b755257496"} Jul 20 22:59:25.674141 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:25.674112 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" event={"ID":"d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2","Type":"ContainerStarted","Data":"ec259887df5eeb4346c2bcdd55ba00d01fcfe4914a4309d2054a43d2ea093331"} Jul 20 22:59:27.061023 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.060982 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm"] Jul 20 22:59:27.061444 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.061193 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" podUID="e32ad569-2394-4b70-a45e-c48fc4c8b00e" containerName="discovery" containerID="cri-o://8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20" gracePeriod=30 Jul 20 22:59:27.483323 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.483292 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:27.559295 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.559266 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-dns-cert\") pod \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " Jul 20 22:59:27.559489 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.559307 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvg4t\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-kube-api-access-wvg4t\") pod \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " Jul 20 22:59:27.559489 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.559340 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-kubeconfig\") pod \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " Jul 20 22:59:27.559595 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.559487 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ad569-2394-4b70-a45e-c48fc4c8b00e-local-certs\") pod \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " Jul 20 22:59:27.559595 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.559535 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-ca-configmap\") pod \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " Jul 20 22:59:27.559595 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.559567 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-cacerts\") pod \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " Jul 20 22:59:27.559595 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.559584 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-token\") pod \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\" (UID: \"e32ad569-2394-4b70-a45e-c48fc4c8b00e\") " Jul 20 22:59:27.560226 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.560196 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e32ad569-2394-4b70-a45e-c48fc4c8b00e" (UID: "e32ad569-2394-4b70-a45e-c48fc4c8b00e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 22:59:27.561904 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.561803 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ad569-2394-4b70-a45e-c48fc4c8b00e-local-certs" (OuterVolumeSpecName: "local-certs") pod "e32ad569-2394-4b70-a45e-c48fc4c8b00e" (UID: "e32ad569-2394-4b70-a45e-c48fc4c8b00e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 22:59:27.561904 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.561860 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-kube-api-access-wvg4t" (OuterVolumeSpecName: "kube-api-access-wvg4t") pod "e32ad569-2394-4b70-a45e-c48fc4c8b00e" (UID: "e32ad569-2394-4b70-a45e-c48fc4c8b00e"). InnerVolumeSpecName "kube-api-access-wvg4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:59:27.562064 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.561919 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e32ad569-2394-4b70-a45e-c48fc4c8b00e" (UID: "e32ad569-2394-4b70-a45e-c48fc4c8b00e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 22:59:27.562064 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.561930 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e32ad569-2394-4b70-a45e-c48fc4c8b00e" (UID: "e32ad569-2394-4b70-a45e-c48fc4c8b00e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 22:59:27.562165 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.562076 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-cacerts" (OuterVolumeSpecName: "cacerts") pod "e32ad569-2394-4b70-a45e-c48fc4c8b00e" (UID: "e32ad569-2394-4b70-a45e-c48fc4c8b00e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 22:59:27.562165 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.562132 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-token" (OuterVolumeSpecName: "istio-token") pod "e32ad569-2394-4b70-a45e-c48fc4c8b00e" (UID: "e32ad569-2394-4b70-a45e-c48fc4c8b00e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:59:27.660480 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.660422 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-ca-configmap\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:59:27.660480 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.660473 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-cacerts\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:59:27.660480 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.660489 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-token\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:59:27.660785 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.660503 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-csr-dns-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:59:27.660785 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.660515 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvg4t\" (UniqueName: \"kubernetes.io/projected/e32ad569-2394-4b70-a45e-c48fc4c8b00e-kube-api-access-wvg4t\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:59:27.660785 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.660531 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e32ad569-2394-4b70-a45e-c48fc4c8b00e-istio-kubeconfig\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:59:27.660785 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.660542 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ad569-2394-4b70-a45e-c48fc4c8b00e-local-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 22:59:27.683685 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.683648 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" event={"ID":"cf87b6dd-57f0-4a91-8750-f7599613751d","Type":"ContainerStarted","Data":"977d7f1ce8346a771eda33a6b9bc2350d83f45846e490272935487c5ad5b9368"} Jul 20 22:59:27.683859 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.683734 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:27.684981 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.684960 2574 generic.go:356] "Generic (PLEG): container finished" podID="e32ad569-2394-4b70-a45e-c48fc4c8b00e" containerID="8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20" exitCode=0 Jul 20 22:59:27.685057 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.684986 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" event={"ID":"e32ad569-2394-4b70-a45e-c48fc4c8b00e","Type":"ContainerDied","Data":"8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20"} Jul 20 22:59:27.685057 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.685035 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" event={"ID":"e32ad569-2394-4b70-a45e-c48fc4c8b00e","Type":"ContainerDied","Data":"ca26f6b6994ffc7f71629273f03366078cff69dd7a0d43bbe3ae0fc1ac6ace3b"} Jul 20 22:59:27.685143 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.685068 2574 scope.go:117] "RemoveContainer" containerID="8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20" Jul 20 22:59:27.685143 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.684999 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm" Jul 20 22:59:27.699126 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.699095 2574 scope.go:117] "RemoveContainer" containerID="8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20" Jul 20 22:59:27.699608 ip-10-0-129-209 kubenswrapper[2574]: E0720 22:59:27.699575 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20\": container with ID starting with 8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20 not found: ID does not exist" containerID="8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20" Jul 20 22:59:27.699679 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.699631 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20"} err="failed to get container status \"8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20\": rpc error: code = NotFound desc = could not find container \"8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20\": container with ID starting with 8b9a49db9886f705eed1fa66bcc96ab8c5a596dfd0f6057b7066f4a2afcd2e20 not found: ID does not exist" Jul 20 22:59:27.702391 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.702350 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" podStartSLOduration=2.380192499 podStartE2EDuration="4.702308001s" totalImagesPullingTime="2.322115502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:27.70134144 +0000 UTC m=+343.019693721" watchObservedRunningTime="2026-07-20 22:59:27.702308001 +0000 UTC m=+343.020660287" Jul 20 22:59:27.717379 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.717350 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm"] Jul 20 22:59:27.721843 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:27.721821 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6mrrm"] Jul 20 22:59:28.690479 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:28.690381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" event={"ID":"d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2","Type":"ContainerStarted","Data":"0d2af14234b433625d6fb7cf31bb121663712df2e9a8f2d1e5cfee7ae22c41ae"} Jul 20 22:59:28.690892 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:28.690516 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 22:59:28.720438 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:28.720389 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" podStartSLOduration=2.393725631 podStartE2EDuration="5.720370762s" totalImagesPullingTime="3.326645131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:28.719554722 +0000 UTC m=+344.037907008" watchObservedRunningTime="2026-07-20 22:59:28.720370762 +0000 UTC m=+344.038723055" Jul 20 22:59:29.226527 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:29.226494 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e32ad569-2394-4b70-a45e-c48fc4c8b00e" path="/var/lib/kubelet/pods/e32ad569-2394-4b70-a45e-c48fc4c8b00e/volumes" Jul 20 22:59:30.659369 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:30.659341 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-tnwnd" Jul 20 22:59:58.696661 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:58.696632 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7t68m" Jul 20 22:59:59.697519 ip-10-0-129-209 kubenswrapper[2574]: I0720 22:59:59.697487 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bn26h" Jul 20 23:00:14.646224 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.646191 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cc8z5"] Jul 20 23:00:14.646677 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.646489 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32ad569-2394-4b70-a45e-c48fc4c8b00e" containerName="discovery" Jul 20 23:00:14.646677 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.646501 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32ad569-2394-4b70-a45e-c48fc4c8b00e" containerName="discovery" Jul 20 23:00:14.646677 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.646559 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e32ad569-2394-4b70-a45e-c48fc4c8b00e" containerName="discovery" Jul 20 23:00:14.649355 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.649338 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" Jul 20 23:00:14.652032 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.652007 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 23:00:14.652032 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.652007 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 23:00:14.652217 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.652048 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-nn5ls\"" Jul 20 23:00:14.665638 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.665607 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cc8z5"] Jul 20 23:00:14.725963 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.725928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqng6\" (UniqueName: \"kubernetes.io/projected/5e515d1f-7b4c-4f6b-9d44-783457953419-kube-api-access-hqng6\") pod \"kuadrant-operator-catalog-cc8z5\" (UID: \"5e515d1f-7b4c-4f6b-9d44-783457953419\") " pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" Jul 20 23:00:14.826888 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.826853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqng6\" (UniqueName: \"kubernetes.io/projected/5e515d1f-7b4c-4f6b-9d44-783457953419-kube-api-access-hqng6\") pod \"kuadrant-operator-catalog-cc8z5\" (UID: \"5e515d1f-7b4c-4f6b-9d44-783457953419\") " pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" Jul 20 23:00:14.835768 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.835740 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqng6\" (UniqueName: \"kubernetes.io/projected/5e515d1f-7b4c-4f6b-9d44-783457953419-kube-api-access-hqng6\") pod \"kuadrant-operator-catalog-cc8z5\" (UID: \"5e515d1f-7b4c-4f6b-9d44-783457953419\") " pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" Jul 20 23:00:14.959472 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:14.959377 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" Jul 20 23:00:15.006799 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.006762 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cc8z5"] Jul 20 23:00:15.094111 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.094086 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cc8z5"] Jul 20 23:00:15.096411 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:00:15.096380 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e515d1f_7b4c_4f6b_9d44_783457953419.slice/crio-d0f2b9e1157ac125a16f830b27ed1920b9e1b7e135162d96d8efa2bdde0168ec WatchSource:0}: Error finding container d0f2b9e1157ac125a16f830b27ed1920b9e1b7e135162d96d8efa2bdde0168ec: Status 404 returned error can't find the container with id d0f2b9e1157ac125a16f830b27ed1920b9e1b7e135162d96d8efa2bdde0168ec Jul 20 23:00:15.216249 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.216174 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9kkp2"] Jul 20 23:00:15.220345 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.220329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" Jul 20 23:00:15.227649 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.227620 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9kkp2"] Jul 20 23:00:15.330586 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.330553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55qj\" (UniqueName: \"kubernetes.io/projected/9555a46d-3328-4463-b43a-36a4ccafd374-kube-api-access-k55qj\") pod \"kuadrant-operator-catalog-9kkp2\" (UID: \"9555a46d-3328-4463-b43a-36a4ccafd374\") " pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" Jul 20 23:00:15.431618 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.431584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55qj\" (UniqueName: \"kubernetes.io/projected/9555a46d-3328-4463-b43a-36a4ccafd374-kube-api-access-k55qj\") pod \"kuadrant-operator-catalog-9kkp2\" (UID: \"9555a46d-3328-4463-b43a-36a4ccafd374\") " pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" Jul 20 23:00:15.439977 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.439946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55qj\" (UniqueName: \"kubernetes.io/projected/9555a46d-3328-4463-b43a-36a4ccafd374-kube-api-access-k55qj\") pod \"kuadrant-operator-catalog-9kkp2\" (UID: \"9555a46d-3328-4463-b43a-36a4ccafd374\") " pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" Jul 20 23:00:15.530875 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.530796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" Jul 20 23:00:15.842448 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.842260 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9kkp2"] Jul 20 23:00:15.852254 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:00:15.852223 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9555a46d_3328_4463_b43a_36a4ccafd374.slice/crio-ead78a1ab512db37746a18b4a57d44a7c42633000d2120edec644b06d2222798 WatchSource:0}: Error finding container ead78a1ab512db37746a18b4a57d44a7c42633000d2120edec644b06d2222798: Status 404 returned error can't find the container with id ead78a1ab512db37746a18b4a57d44a7c42633000d2120edec644b06d2222798 Jul 20 23:00:15.860783 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.860745 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" event={"ID":"9555a46d-3328-4463-b43a-36a4ccafd374","Type":"ContainerStarted","Data":"ead78a1ab512db37746a18b4a57d44a7c42633000d2120edec644b06d2222798"} Jul 20 23:00:15.861964 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:15.861939 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" event={"ID":"5e515d1f-7b4c-4f6b-9d44-783457953419","Type":"ContainerStarted","Data":"d0f2b9e1157ac125a16f830b27ed1920b9e1b7e135162d96d8efa2bdde0168ec"} Jul 20 23:00:17.881967 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:17.881927 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" event={"ID":"9555a46d-3328-4463-b43a-36a4ccafd374","Type":"ContainerStarted","Data":"cf72f60a652f48442d9e8a9e21f61fae970cdda648d56c7a785ce507b85e1826"} Jul 20 23:00:17.883376 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:17.883348 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" event={"ID":"5e515d1f-7b4c-4f6b-9d44-783457953419","Type":"ContainerStarted","Data":"337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7"} Jul 20 23:00:17.883565 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:17.883430 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" podUID="5e515d1f-7b4c-4f6b-9d44-783457953419" containerName="registry-server" containerID="cri-o://337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7" gracePeriod=2 Jul 20 23:00:17.898079 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:17.898023 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" podStartSLOduration=1.4878514649999999 podStartE2EDuration="2.898002913s" totalImagesPullingTime="1.410151448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:00:17.897771053 +0000 UTC m=+393.216123338" watchObservedRunningTime="2026-07-20 23:00:17.898002913 +0000 UTC m=+393.216355213" Jul 20 23:00:17.914091 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:17.914039 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" podStartSLOduration=1.750809775 podStartE2EDuration="3.914022971s" totalImagesPullingTime="2.163213196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:00:17.913745505 +0000 UTC m=+393.232097793" watchObservedRunningTime="2026-07-20 23:00:17.914022971 +0000 UTC m=+393.232375256" Jul 20 23:00:18.120316 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.120291 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" Jul 20 23:00:18.254491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.254366 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqng6\" (UniqueName: \"kubernetes.io/projected/5e515d1f-7b4c-4f6b-9d44-783457953419-kube-api-access-hqng6\") pod \"5e515d1f-7b4c-4f6b-9d44-783457953419\" (UID: \"5e515d1f-7b4c-4f6b-9d44-783457953419\") " Jul 20 23:00:18.256528 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.256499 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e515d1f-7b4c-4f6b-9d44-783457953419-kube-api-access-hqng6" (OuterVolumeSpecName: "kube-api-access-hqng6") pod "5e515d1f-7b4c-4f6b-9d44-783457953419" (UID: "5e515d1f-7b4c-4f6b-9d44-783457953419"). InnerVolumeSpecName "kube-api-access-hqng6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:00:18.355471 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.355437 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqng6\" (UniqueName: \"kubernetes.io/projected/5e515d1f-7b4c-4f6b-9d44-783457953419-kube-api-access-hqng6\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:00:18.888646 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.888615 2574 generic.go:356] "Generic (PLEG): container finished" podID="5e515d1f-7b4c-4f6b-9d44-783457953419" containerID="337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7" exitCode=0 Jul 20 23:00:18.889046 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.888662 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" Jul 20 23:00:18.889046 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.888708 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" event={"ID":"5e515d1f-7b4c-4f6b-9d44-783457953419","Type":"ContainerDied","Data":"337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7"} Jul 20 23:00:18.889046 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.888753 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cc8z5" event={"ID":"5e515d1f-7b4c-4f6b-9d44-783457953419","Type":"ContainerDied","Data":"d0f2b9e1157ac125a16f830b27ed1920b9e1b7e135162d96d8efa2bdde0168ec"} Jul 20 23:00:18.889046 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.888778 2574 scope.go:117] "RemoveContainer" containerID="337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7" Jul 20 23:00:18.897984 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.897965 2574 scope.go:117] "RemoveContainer" containerID="337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7" Jul 20 23:00:18.898224 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:00:18.898207 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7\": container with ID starting with 337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7 not found: ID does not exist" containerID="337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7" Jul 20 23:00:18.898279 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.898235 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7"} err="failed to get container status \"337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7\": rpc error: code = NotFound desc = could not find container \"337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7\": container with ID starting with 337f8c39df1188576644f01ba5e64304d2305d80eb87ee4f7007ea18d07d78b7 not found: ID does not exist" Jul 20 23:00:18.910417 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.910387 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cc8z5"] Jul 20 23:00:18.912969 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:18.912949 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cc8z5"] Jul 20 23:00:19.227658 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:19.227566 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e515d1f-7b4c-4f6b-9d44-783457953419" path="/var/lib/kubelet/pods/5e515d1f-7b4c-4f6b-9d44-783457953419/volumes" Jul 20 23:00:25.531477 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:25.531426 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" Jul 20 23:00:25.531889 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:25.531525 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" Jul 20 23:00:25.553949 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:25.553921 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" Jul 20 23:00:25.939767 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:25.939735 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-9kkp2" Jul 20 23:00:48.145898 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.145864 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9"] Jul 20 23:00:48.146327 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.146145 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e515d1f-7b4c-4f6b-9d44-783457953419" containerName="registry-server" Jul 20 23:00:48.146327 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.146157 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e515d1f-7b4c-4f6b-9d44-783457953419" containerName="registry-server" Jul 20 23:00:48.146327 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.146205 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e515d1f-7b4c-4f6b-9d44-783457953419" containerName="registry-server" Jul 20 23:00:48.150316 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.150281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:00:48.153214 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.153192 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2s4lg\"" Jul 20 23:00:48.164961 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.164933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9"] Jul 20 23:00:48.189535 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.189491 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" (UID: \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:00:48.189700 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.189560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prvg\" (UniqueName: \"kubernetes.io/projected/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-kube-api-access-7prvg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" (UID: \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:00:48.290835 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.290796 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prvg\" (UniqueName: \"kubernetes.io/projected/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-kube-api-access-7prvg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" (UID: \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:00:48.291041 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.290885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" (UID: \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:00:48.291303 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.291279 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" (UID: \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:00:48.300362 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.300332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prvg\" (UniqueName: \"kubernetes.io/projected/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-kube-api-access-7prvg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" (UID: \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:00:48.461361 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.461282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:00:48.606419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:48.606391 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9"] Jul 20 23:00:48.608922 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:00:48.608893 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b8312f_2b44_4d9d_8a40_9921bd37a88d.slice/crio-2eb06c567888698d9cbc32a9c2051fbfea01761f28a45d23f74303e1829a6bfe WatchSource:0}: Error finding container 2eb06c567888698d9cbc32a9c2051fbfea01761f28a45d23f74303e1829a6bfe: Status 404 returned error can't find the container with id 2eb06c567888698d9cbc32a9c2051fbfea01761f28a45d23f74303e1829a6bfe Jul 20 23:00:49.009561 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:49.009526 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" event={"ID":"d4b8312f-2b44-4d9d-8a40-9921bd37a88d","Type":"ContainerStarted","Data":"2eb06c567888698d9cbc32a9c2051fbfea01761f28a45d23f74303e1829a6bfe"} Jul 20 23:00:50.067508 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:50.067470 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49"] Jul 20 23:00:50.070729 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:50.070708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:00:50.073371 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:50.073342 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2rzml\"" Jul 20 23:00:50.080170 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:50.080142 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49"] Jul 20 23:00:50.105801 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:50.105760 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssc2\" (UniqueName: \"kubernetes.io/projected/4b742e31-122d-4f24-b595-d57ab9c8ed06-kube-api-access-bssc2\") pod \"limitador-operator-controller-manager-859d7b78bd-f4g49\" (UID: \"4b742e31-122d-4f24-b595-d57ab9c8ed06\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:00:50.207094 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:50.207057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssc2\" (UniqueName: \"kubernetes.io/projected/4b742e31-122d-4f24-b595-d57ab9c8ed06-kube-api-access-bssc2\") pod \"limitador-operator-controller-manager-859d7b78bd-f4g49\" (UID: \"4b742e31-122d-4f24-b595-d57ab9c8ed06\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:00:50.223386 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:50.223352 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssc2\" (UniqueName: \"kubernetes.io/projected/4b742e31-122d-4f24-b595-d57ab9c8ed06-kube-api-access-bssc2\") pod \"limitador-operator-controller-manager-859d7b78bd-f4g49\" (UID: \"4b742e31-122d-4f24-b595-d57ab9c8ed06\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:00:50.381194 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:50.381150 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:00:50.533371 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:50.533343 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49"] Jul 20 23:00:50.535858 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:00:50.535821 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b742e31_122d_4f24_b595_d57ab9c8ed06.slice/crio-ca126694b67524336a2451e4621825770337e20db6cf95ffee6e3f20d9d3bbd3 WatchSource:0}: Error finding container ca126694b67524336a2451e4621825770337e20db6cf95ffee6e3f20d9d3bbd3: Status 404 returned error can't find the container with id ca126694b67524336a2451e4621825770337e20db6cf95ffee6e3f20d9d3bbd3 Jul 20 23:00:51.018584 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:51.018553 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" event={"ID":"4b742e31-122d-4f24-b595-d57ab9c8ed06","Type":"ContainerStarted","Data":"ca126694b67524336a2451e4621825770337e20db6cf95ffee6e3f20d9d3bbd3"} Jul 20 23:00:52.584100 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:52.584061 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj"] Jul 20 23:00:52.587387 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:52.587364 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" Jul 20 23:00:52.591314 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:52.591297 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xqgb4\"" Jul 20 23:00:52.602016 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:52.601991 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj"] Jul 20 23:00:52.626967 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:52.626935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnwn\" (UniqueName: \"kubernetes.io/projected/eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a-kube-api-access-7dnwn\") pod \"authorino-operator-75cc8d5cb7-tvhxj\" (UID: \"eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" Jul 20 23:00:52.727633 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:52.727591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnwn\" (UniqueName: \"kubernetes.io/projected/eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a-kube-api-access-7dnwn\") pod \"authorino-operator-75cc8d5cb7-tvhxj\" (UID: \"eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" Jul 20 23:00:52.738967 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:52.738933 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnwn\" (UniqueName: \"kubernetes.io/projected/eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a-kube-api-access-7dnwn\") pod \"authorino-operator-75cc8d5cb7-tvhxj\" (UID: \"eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" Jul 20 23:00:52.897206 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:52.897165 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" Jul 20 23:00:53.029738 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:53.029717 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj"] Jul 20 23:00:53.032062 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:53.032029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" event={"ID":"4b742e31-122d-4f24-b595-d57ab9c8ed06","Type":"ContainerStarted","Data":"e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab"} Jul 20 23:00:53.032211 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:53.032114 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:00:53.032488 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:00:53.032449 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb043f2e_50ab_49d9_bfe6_1a6c960ebe6a.slice/crio-4b920604f1b36ad891436dd9e5912cf330ae51222081701be33f7fb90f07c136 WatchSource:0}: Error finding container 4b920604f1b36ad891436dd9e5912cf330ae51222081701be33f7fb90f07c136: Status 404 returned error can't find the container with id 4b920604f1b36ad891436dd9e5912cf330ae51222081701be33f7fb90f07c136 Jul 20 23:00:53.049821 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:53.049774 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" podStartSLOduration=1.530184285 podStartE2EDuration="3.049757641s" totalImagesPullingTime="1.519573356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:00:53.049367285 +0000 UTC m=+428.367719575" watchObservedRunningTime="2026-07-20 23:00:53.049757641 +0000 UTC m=+428.368109926" Jul 20 23:00:54.038512 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:54.038471 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" event={"ID":"eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a","Type":"ContainerStarted","Data":"4b920604f1b36ad891436dd9e5912cf330ae51222081701be33f7fb90f07c136"} Jul 20 23:00:55.044897 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:55.044813 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" event={"ID":"d4b8312f-2b44-4d9d-8a40-9921bd37a88d","Type":"ContainerStarted","Data":"dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383"} Jul 20 23:00:55.045337 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:55.044957 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:00:55.075810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:55.075746 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" podStartSLOduration=0.932538041 podStartE2EDuration="7.075724008s" totalImagesPullingTime="6.143185967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:00:55.073847215 +0000 UTC m=+430.392199503" watchObservedRunningTime="2026-07-20 23:00:55.075724008 +0000 UTC m=+430.394076293" Jul 20 23:00:57.054810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.054775 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" event={"ID":"eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a","Type":"ContainerStarted","Data":"182991c277ff1b2f4c39d60560da855caad82b384cd73cd61d466f3d90a7e1a0"} Jul 20 23:00:57.055293 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.054905 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" Jul 20 23:00:57.077765 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.077710 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" podStartSLOduration=1.66406519 podStartE2EDuration="5.077693968s" totalImagesPullingTime="3.413628778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:00:57.076517707 +0000 UTC m=+432.394869996" watchObservedRunningTime="2026-07-20 23:00:57.077693968 +0000 UTC m=+432.396046254" Jul 20 23:00:57.791997 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.791959 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx"] Jul 20 23:00:57.795332 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.795310 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:57.798166 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.798138 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 20 23:00:57.798166 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.798160 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hqlpd\"" Jul 20 23:00:57.798356 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.798257 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 20 23:00:57.803780 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.803750 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx"] Jul 20 23:00:57.872809 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.872767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/17011a37-8053-49ef-a6a2-23cee3a83dd9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mqrvx\" (UID: \"17011a37-8053-49ef-a6a2-23cee3a83dd9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:57.872809 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.872811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/17011a37-8053-49ef-a6a2-23cee3a83dd9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-mqrvx\" (UID: \"17011a37-8053-49ef-a6a2-23cee3a83dd9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:57.873051 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.872906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66cqd\" (UniqueName: \"kubernetes.io/projected/17011a37-8053-49ef-a6a2-23cee3a83dd9-kube-api-access-66cqd\") pod \"kuadrant-console-plugin-66b5774985-mqrvx\" (UID: \"17011a37-8053-49ef-a6a2-23cee3a83dd9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:57.974286 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.974240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66cqd\" (UniqueName: \"kubernetes.io/projected/17011a37-8053-49ef-a6a2-23cee3a83dd9-kube-api-access-66cqd\") pod \"kuadrant-console-plugin-66b5774985-mqrvx\" (UID: \"17011a37-8053-49ef-a6a2-23cee3a83dd9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:57.974484 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.974300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/17011a37-8053-49ef-a6a2-23cee3a83dd9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mqrvx\" (UID: \"17011a37-8053-49ef-a6a2-23cee3a83dd9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:57.974484 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.974321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/17011a37-8053-49ef-a6a2-23cee3a83dd9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-mqrvx\" (UID: \"17011a37-8053-49ef-a6a2-23cee3a83dd9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:57.975000 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.974981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/17011a37-8053-49ef-a6a2-23cee3a83dd9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-mqrvx\" (UID: \"17011a37-8053-49ef-a6a2-23cee3a83dd9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:57.976741 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.976719 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/17011a37-8053-49ef-a6a2-23cee3a83dd9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mqrvx\" (UID: \"17011a37-8053-49ef-a6a2-23cee3a83dd9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:57.984810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:57.984786 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66cqd\" (UniqueName: \"kubernetes.io/projected/17011a37-8053-49ef-a6a2-23cee3a83dd9-kube-api-access-66cqd\") pod \"kuadrant-console-plugin-66b5774985-mqrvx\" (UID: \"17011a37-8053-49ef-a6a2-23cee3a83dd9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:58.104900 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:58.104874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" Jul 20 23:00:58.230802 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:58.230777 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx"] Jul 20 23:00:58.232812 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:00:58.232780 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17011a37_8053_49ef_a6a2_23cee3a83dd9.slice/crio-8d3daad29105c21ffd5d3fa2d1b3656ac13f0d266f4d298751f6612926344934 WatchSource:0}: Error finding container 8d3daad29105c21ffd5d3fa2d1b3656ac13f0d266f4d298751f6612926344934: Status 404 returned error can't find the container with id 8d3daad29105c21ffd5d3fa2d1b3656ac13f0d266f4d298751f6612926344934 Jul 20 23:00:59.065747 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:00:59.065695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" event={"ID":"17011a37-8053-49ef-a6a2-23cee3a83dd9","Type":"ContainerStarted","Data":"8d3daad29105c21ffd5d3fa2d1b3656ac13f0d266f4d298751f6612926344934"} Jul 20 23:01:04.041414 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:04.041381 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:06.053048 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:06.052999 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:07.721271 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.721185 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9"] Jul 20 23:01:07.721728 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.721425 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" containerName="manager" containerID="cri-o://dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383" gracePeriod=2 Jul 20 23:01:07.726137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.726071 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9"] Jul 20 23:01:07.745780 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.745721 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49"] Jul 20 23:01:07.746889 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.746075 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" containerName="manager" containerID="cri-o://e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab" gracePeriod=2 Jul 20 23:01:07.746889 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.746662 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49"] Jul 20 23:01:07.751734 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:07.751650 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:07.754327 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:07.754288 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:07.756782 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.756755 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx"] Jul 20 23:01:07.757224 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.757201 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" containerName="manager" Jul 20 23:01:07.757224 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.757226 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" containerName="manager" Jul 20 23:01:07.757390 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.757247 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" containerName="manager" Jul 20 23:01:07.757390 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.757256 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" containerName="manager" Jul 20 23:01:07.757390 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.757326 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" containerName="manager" Jul 20 23:01:07.757390 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.757336 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" containerName="manager" Jul 20 23:01:07.760627 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.760603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:07.763392 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:07.763356 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:07.779355 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.779276 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx"] Jul 20 23:01:07.794278 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.794238 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq"] Jul 20 23:01:07.799415 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.799390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" Jul 20 23:01:07.812389 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:07.812344 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:07.814774 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.814740 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq"] Jul 20 23:01:07.814898 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:07.814853 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:07.844261 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:07.843929 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:07.869689 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.869655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm452\" (UniqueName: \"kubernetes.io/projected/b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9-kube-api-access-gm452\") pod \"limitador-operator-controller-manager-859d7b78bd-x92zq\" (UID: \"b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" Jul 20 23:01:07.869849 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.869753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4mk5\" (UniqueName: \"kubernetes.io/projected/77803fd5-7afa-4fcf-befd-30157fc52875-kube-api-access-k4mk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jl7tx\" (UID: \"77803fd5-7afa-4fcf-befd-30157fc52875\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:07.869849 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.869819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77803fd5-7afa-4fcf-befd-30157fc52875-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jl7tx\" (UID: \"77803fd5-7afa-4fcf-befd-30157fc52875\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:07.971079 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.971038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm452\" (UniqueName: \"kubernetes.io/projected/b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9-kube-api-access-gm452\") pod \"limitador-operator-controller-manager-859d7b78bd-x92zq\" (UID: \"b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" Jul 20 23:01:07.971268 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.971120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4mk5\" (UniqueName: \"kubernetes.io/projected/77803fd5-7afa-4fcf-befd-30157fc52875-kube-api-access-k4mk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jl7tx\" (UID: \"77803fd5-7afa-4fcf-befd-30157fc52875\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:07.971268 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.971151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77803fd5-7afa-4fcf-befd-30157fc52875-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jl7tx\" (UID: \"77803fd5-7afa-4fcf-befd-30157fc52875\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:07.971628 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.971558 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77803fd5-7afa-4fcf-befd-30157fc52875-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jl7tx\" (UID: \"77803fd5-7afa-4fcf-befd-30157fc52875\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:07.981288 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.981245 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4mk5\" (UniqueName: \"kubernetes.io/projected/77803fd5-7afa-4fcf-befd-30157fc52875-kube-api-access-k4mk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jl7tx\" (UID: \"77803fd5-7afa-4fcf-befd-30157fc52875\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:07.981476 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:07.981365 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm452\" (UniqueName: \"kubernetes.io/projected/b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9-kube-api-access-gm452\") pod \"limitador-operator-controller-manager-859d7b78bd-x92zq\" (UID: \"b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" Jul 20 23:01:08.061946 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:08.061915 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tvhxj" Jul 20 23:01:08.084116 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:08.084064 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:08.086158 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:08.086107 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:08.161281 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:08.161246 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:08.170760 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:08.170724 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" Jul 20 23:01:20.057366 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.057337 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:20.059901 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.059870 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:20.069922 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.069898 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:20.072341 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.072312 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:20.074580 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.074555 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:20.167890 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.167793 2574 generic.go:356] "Generic (PLEG): container finished" podID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" containerID="dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383" exitCode=0 Jul 20 23:01:20.167890 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.167833 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:20.167890 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.167874 2574 scope.go:117] "RemoveContainer" containerID="dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383" Jul 20 23:01:20.169254 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.169233 2574 generic.go:356] "Generic (PLEG): container finished" podID="4b742e31-122d-4f24-b595-d57ab9c8ed06" containerID="e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab" exitCode=0 Jul 20 23:01:20.169359 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.169272 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:20.170390 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.170360 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:20.171141 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.171109 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" event={"ID":"17011a37-8053-49ef-a6a2-23cee3a83dd9","Type":"ContainerStarted","Data":"c78568e3fb52db82f6688b4eb4d07b91831e1eed9d1259f5b15432c367addc13"} Jul 20 23:01:20.172548 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.172522 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:20.174659 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.174635 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:20.176258 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.176246 2574 scope.go:117] "RemoveContainer" containerID="dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383" Jul 20 23:01:20.176557 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.176534 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383\": container with ID starting with dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383 not found: ID does not exist" containerID="dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383" Jul 20 23:01:20.176663 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.176565 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383"} err="failed to get container status \"dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383\": rpc error: code = NotFound desc = could not find container \"dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383\": container with ID starting with dc1084fd5aee1b86dc7a6f68533b22ccfb8de133da9aed11bf4051d586337383 not found: ID does not exist" Jul 20 23:01:20.176663 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.176583 2574 scope.go:117] "RemoveContainer" containerID="e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab" Jul 20 23:01:20.176663 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.176622 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:20.183935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.183914 2574 scope.go:117] "RemoveContainer" containerID="e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab" Jul 20 23:01:20.184102 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.184082 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-extensions-socket-volume\") pod \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\" (UID: \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\") " Jul 20 23:01:20.184173 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.184160 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7prvg\" (UniqueName: \"kubernetes.io/projected/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-kube-api-access-7prvg\") pod \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\" (UID: \"d4b8312f-2b44-4d9d-8a40-9921bd37a88d\") " Jul 20 23:01:20.184216 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.184179 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bssc2\" (UniqueName: \"kubernetes.io/projected/4b742e31-122d-4f24-b595-d57ab9c8ed06-kube-api-access-bssc2\") pod \"4b742e31-122d-4f24-b595-d57ab9c8ed06\" (UID: \"4b742e31-122d-4f24-b595-d57ab9c8ed06\") " Jul 20 23:01:20.184216 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.184184 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab\": container with ID starting with e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab not found: ID does not exist" containerID="e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab" Jul 20 23:01:20.184300 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.184214 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab"} err="failed to get container status \"e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab\": rpc error: code = NotFound desc = could not find container \"e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab\": container with ID starting with e0c51f2817914db56a3ffcd213933fbe97b80d6425125732e0266dc6ae1250ab not found: ID does not exist" Jul 20 23:01:20.184747 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.184722 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d4b8312f-2b44-4d9d-8a40-9921bd37a88d" (UID: "d4b8312f-2b44-4d9d-8a40-9921bd37a88d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:01:20.186104 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.186077 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-kube-api-access-7prvg" (OuterVolumeSpecName: "kube-api-access-7prvg") pod "d4b8312f-2b44-4d9d-8a40-9921bd37a88d" (UID: "d4b8312f-2b44-4d9d-8a40-9921bd37a88d"). InnerVolumeSpecName "kube-api-access-7prvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:01:20.186188 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.186101 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b742e31-122d-4f24-b595-d57ab9c8ed06-kube-api-access-bssc2" (OuterVolumeSpecName: "kube-api-access-bssc2") pod "4b742e31-122d-4f24-b595-d57ab9c8ed06" (UID: "4b742e31-122d-4f24-b595-d57ab9c8ed06"). InnerVolumeSpecName "kube-api-access-bssc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:01:20.192068 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.192017 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mqrvx" podStartSLOduration=1.429587162 podStartE2EDuration="23.192002075s" totalImagesPullingTime="21.762414913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:01:20.191086815 +0000 UTC m=+455.509439100" watchObservedRunningTime="2026-07-20 23:01:20.192002075 +0000 UTC m=+455.510354362" Jul 20 23:01:20.269342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.269310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq"] Jul 20 23:01:20.271698 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.271674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx"] Jul 20 23:01:20.272746 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:01:20.272724 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb318e2e7_65e8_4a2e_8b33_a1a2cdee35b9.slice/crio-e84ebfde295fb9267d5fe3e8d73ee4e7849c72ef1fb89a95b72c9d5b2b2ce7b5 WatchSource:0}: Error finding container e84ebfde295fb9267d5fe3e8d73ee4e7849c72ef1fb89a95b72c9d5b2b2ce7b5: Status 404 returned error can't find the container with id e84ebfde295fb9267d5fe3e8d73ee4e7849c72ef1fb89a95b72c9d5b2b2ce7b5 Jul 20 23:01:20.274118 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:01:20.274091 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77803fd5_7afa_4fcf_befd_30157fc52875.slice/crio-6c072faf53110fd643915bd19bdcac45b0d18c37e06437b0bf1544a06d2bc2ed WatchSource:0}: Error finding container 6c072faf53110fd643915bd19bdcac45b0d18c37e06437b0bf1544a06d2bc2ed: Status 404 returned error can't find the container with id 6c072faf53110fd643915bd19bdcac45b0d18c37e06437b0bf1544a06d2bc2ed Jul 20 23:01:20.284816 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.284795 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-extensions-socket-volume\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:01:20.284918 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.284818 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7prvg\" (UniqueName: \"kubernetes.io/projected/d4b8312f-2b44-4d9d-8a40-9921bd37a88d-kube-api-access-7prvg\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:01:20.284918 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:20.284831 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bssc2\" (UniqueName: \"kubernetes.io/projected/4b742e31-122d-4f24-b595-d57ab9c8ed06-kube-api-access-bssc2\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:01:20.481934 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.481903 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:20.484769 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.484727 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:20.487284 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.487245 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:20.489934 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:20.489903 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:21.177286 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.177201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" event={"ID":"77803fd5-7afa-4fcf-befd-30157fc52875","Type":"ContainerStarted","Data":"e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c"} Jul 20 23:01:21.177286 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.177256 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" event={"ID":"77803fd5-7afa-4fcf-befd-30157fc52875","Type":"ContainerStarted","Data":"6c072faf53110fd643915bd19bdcac45b0d18c37e06437b0bf1544a06d2bc2ed"} Jul 20 23:01:21.177765 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.177295 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:21.178684 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.178662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" event={"ID":"b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9","Type":"ContainerStarted","Data":"ff1fdcecf21094a8bbc3db49f30b54301167157fda5a5b117449f5b0d065f459"} Jul 20 23:01:21.178810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.178698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" event={"ID":"b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9","Type":"ContainerStarted","Data":"e84ebfde295fb9267d5fe3e8d73ee4e7849c72ef1fb89a95b72c9d5b2b2ce7b5"} Jul 20 23:01:21.178810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.178718 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" Jul 20 23:01:21.197645 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:21.197602 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:21.198406 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.198364 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" podStartSLOduration=14.198350612 podStartE2EDuration="14.198350612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:01:21.195399714 +0000 UTC m=+456.513752000" watchObservedRunningTime="2026-07-20 23:01:21.198350612 +0000 UTC m=+456.516702898" Jul 20 23:01:21.199685 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:21.199665 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:21.201685 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:21.201666 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4g49\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4g49" Jul 20 23:01:21.203695 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:21.203669 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4bgh9\" is forbidden: User \"system:node:ip-10-0-129-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-209.ec2.internal' and this object" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4bgh9" Jul 20 23:01:21.220804 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.220755 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" podStartSLOduration=14.220738904 podStartE2EDuration="14.220738904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:01:21.219589047 +0000 UTC m=+456.537941332" watchObservedRunningTime="2026-07-20 23:01:21.220738904 +0000 UTC m=+456.539091190" Jul 20 23:01:21.228509 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.228446 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b742e31-122d-4f24-b595-d57ab9c8ed06" path="/var/lib/kubelet/pods/4b742e31-122d-4f24-b595-d57ab9c8ed06/volumes" Jul 20 23:01:21.228908 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:21.228889 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b8312f-2b44-4d9d-8a40-9921bd37a88d" path="/var/lib/kubelet/pods/d4b8312f-2b44-4d9d-8a40-9921bd37a88d/volumes" Jul 20 23:01:25.562392 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:25.562361 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx"] Jul 20 23:01:25.562838 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:25.562557 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" podUID="77803fd5-7afa-4fcf-befd-30157fc52875" containerName="manager" containerID="cri-o://e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c" gracePeriod=10 Jul 20 23:01:25.564412 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:25.564389 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:25.831853 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:25.831831 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:25.934292 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:25.934257 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4mk5\" (UniqueName: \"kubernetes.io/projected/77803fd5-7afa-4fcf-befd-30157fc52875-kube-api-access-k4mk5\") pod \"77803fd5-7afa-4fcf-befd-30157fc52875\" (UID: \"77803fd5-7afa-4fcf-befd-30157fc52875\") " Jul 20 23:01:25.934482 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:25.934332 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77803fd5-7afa-4fcf-befd-30157fc52875-extensions-socket-volume\") pod \"77803fd5-7afa-4fcf-befd-30157fc52875\" (UID: \"77803fd5-7afa-4fcf-befd-30157fc52875\") " Jul 20 23:01:25.934739 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:25.934712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77803fd5-7afa-4fcf-befd-30157fc52875-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "77803fd5-7afa-4fcf-befd-30157fc52875" (UID: "77803fd5-7afa-4fcf-befd-30157fc52875"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:01:25.936247 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:25.936220 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77803fd5-7afa-4fcf-befd-30157fc52875-kube-api-access-k4mk5" (OuterVolumeSpecName: "kube-api-access-k4mk5") pod "77803fd5-7afa-4fcf-befd-30157fc52875" (UID: "77803fd5-7afa-4fcf-befd-30157fc52875"). InnerVolumeSpecName "kube-api-access-k4mk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:01:26.035907 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.035868 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4mk5\" (UniqueName: \"kubernetes.io/projected/77803fd5-7afa-4fcf-befd-30157fc52875-kube-api-access-k4mk5\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:01:26.035907 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.035901 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77803fd5-7afa-4fcf-befd-30157fc52875-extensions-socket-volume\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:01:26.203951 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.203917 2574 generic.go:356] "Generic (PLEG): container finished" podID="77803fd5-7afa-4fcf-befd-30157fc52875" containerID="e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c" exitCode=0 Jul 20 23:01:26.204115 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.203959 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" Jul 20 23:01:26.204115 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.203975 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" event={"ID":"77803fd5-7afa-4fcf-befd-30157fc52875","Type":"ContainerDied","Data":"e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c"} Jul 20 23:01:26.204115 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.204028 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx" event={"ID":"77803fd5-7afa-4fcf-befd-30157fc52875","Type":"ContainerDied","Data":"6c072faf53110fd643915bd19bdcac45b0d18c37e06437b0bf1544a06d2bc2ed"} Jul 20 23:01:26.204115 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.204057 2574 scope.go:117] "RemoveContainer" containerID="e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c" Jul 20 23:01:26.211945 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.211927 2574 scope.go:117] "RemoveContainer" containerID="e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c" Jul 20 23:01:26.212226 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:01:26.212208 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c\": container with ID starting with e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c not found: ID does not exist" containerID="e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c" Jul 20 23:01:26.212302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.212235 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c"} err="failed to get container status \"e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c\": rpc error: code = NotFound desc = could not find container \"e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c\": container with ID starting with e128c20ed95109b4920009ed350893194a1e35d6a1d5c8fcad6d7add6c99790c not found: ID does not exist" Jul 20 23:01:26.225026 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.224999 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx"] Jul 20 23:01:26.233158 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:26.233134 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jl7tx"] Jul 20 23:01:27.226651 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:27.226621 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77803fd5-7afa-4fcf-befd-30157fc52875" path="/var/lib/kubelet/pods/77803fd5-7afa-4fcf-befd-30157fc52875/volumes" Jul 20 23:01:32.188232 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:32.188204 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x92zq" Jul 20 23:01:41.919009 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:41.918969 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4"] Jul 20 23:01:41.920072 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:41.920047 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77803fd5-7afa-4fcf-befd-30157fc52875" containerName="manager" Jul 20 23:01:41.920072 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:41.920071 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="77803fd5-7afa-4fcf-befd-30157fc52875" containerName="manager" Jul 20 23:01:41.920219 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:41.920123 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="77803fd5-7afa-4fcf-befd-30157fc52875" containerName="manager" Jul 20 23:01:41.966301 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:41.966257 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4"] Jul 20 23:01:41.966478 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:41.966373 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:41.969574 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:41.969548 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 23:01:41.969687 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:41.969638 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hz5zz\"" Jul 20 23:01:42.061322 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.061281 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.061322 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.061319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg8pl\" (UniqueName: \"kubernetes.io/projected/5bf0978f-511a-4503-a7c0-8321d105b86f-kube-api-access-lg8pl\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.061569 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.061343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.061569 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.061364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf0978f-511a-4503-a7c0-8321d105b86f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.061569 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.061491 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.061680 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.061577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.061680 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.061605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.061680 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.061623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.061680 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.061654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162220 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162183 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162220 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf0978f-511a-4503-a7c0-8321d105b86f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162393 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162243 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162393 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162508 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162508 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162423 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162508 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162655 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162738 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162817 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162871 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162871 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg8pl\" (UniqueName: \"kubernetes.io/projected/5bf0978f-511a-4503-a7c0-8321d105b86f-kube-api-access-lg8pl\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.162956 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.162882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.163058 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.163033 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf0978f-511a-4503-a7c0-8321d105b86f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.164899 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.164870 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.165026 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.165005 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.171024 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.170970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf0978f-511a-4503-a7c0-8321d105b86f-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.171099 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.171034 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg8pl\" (UniqueName: \"kubernetes.io/projected/5bf0978f-511a-4503-a7c0-8321d105b86f-kube-api-access-lg8pl\") pod \"maas-default-gateway-openshift-default-5896d4dd49-49zl4\" (UID: \"5bf0978f-511a-4503-a7c0-8321d105b86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.277080 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.277048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:42.498980 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:42.498820 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4"] Jul 20 23:01:42.501560 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:01:42.501531 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf0978f_511a_4503_a7c0_8321d105b86f.slice/crio-b797c2bfc2e36e3a58f4a6375ca669612146f812854ae5c75ef57d93d7a279c9 WatchSource:0}: Error finding container b797c2bfc2e36e3a58f4a6375ca669612146f812854ae5c75ef57d93d7a279c9: Status 404 returned error can't find the container with id b797c2bfc2e36e3a58f4a6375ca669612146f812854ae5c75ef57d93d7a279c9 Jul 20 23:01:43.277526 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:43.277485 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" event={"ID":"5bf0978f-511a-4503-a7c0-8321d105b86f","Type":"ContainerStarted","Data":"b797c2bfc2e36e3a58f4a6375ca669612146f812854ae5c75ef57d93d7a279c9"} Jul 20 23:01:46.048026 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:46.047988 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:01:46.048300 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:46.048067 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:01:46.048300 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:46.048098 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:01:46.295206 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:46.295124 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" event={"ID":"5bf0978f-511a-4503-a7c0-8321d105b86f","Type":"ContainerStarted","Data":"56e3c57886e97aec262294acae6a4eb16c47640a8af394a9c2774b6b2cf28d62"} Jul 20 23:01:46.315860 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:46.315801 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" podStartSLOduration=1.77185678 podStartE2EDuration="5.315784688s" totalImagesPullingTime="3.543927908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:01:46.314302922 +0000 UTC m=+481.632655219" watchObservedRunningTime="2026-07-20 23:01:46.315784688 +0000 UTC m=+481.634136973" Jul 20 23:01:47.277964 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.277917 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:47.282764 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.282738 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:47.300265 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.300240 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:47.301265 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.301242 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:01:47.311009 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.310983 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5tmtk"] Jul 20 23:01:47.314385 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.314369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:01:47.317008 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.316992 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 23:01:47.323020 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.322996 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5tmtk"] Jul 20 23:01:47.393772 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.393741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5tmtk"] Jul 20 23:01:47.411603 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.411574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-config-file\") pod \"limitador-limitador-64c94fd64f-5tmtk\" (UID: \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:01:47.411603 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.411612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bq94\" (UniqueName: \"kubernetes.io/projected/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-kube-api-access-5bq94\") pod \"limitador-limitador-64c94fd64f-5tmtk\" (UID: \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:01:47.512838 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.512801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-config-file\") pod \"limitador-limitador-64c94fd64f-5tmtk\" (UID: \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:01:47.512838 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.512840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bq94\" (UniqueName: \"kubernetes.io/projected/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-kube-api-access-5bq94\") pod \"limitador-limitador-64c94fd64f-5tmtk\" (UID: \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:01:47.513439 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.513419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-config-file\") pod \"limitador-limitador-64c94fd64f-5tmtk\" (UID: \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:01:47.521075 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.521043 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bq94\" (UniqueName: \"kubernetes.io/projected/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-kube-api-access-5bq94\") pod \"limitador-limitador-64c94fd64f-5tmtk\" (UID: \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:01:47.624752 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.624716 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:01:47.759608 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:47.759583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5tmtk"] Jul 20 23:01:47.762010 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:01:47.761976 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019a75c6_e42f_49b1_bcfd_efc5b5f199c3.slice/crio-2b5645c12b557816f47b8efeccc9ebdb87c644104cac64215adebf545ef9cf89 WatchSource:0}: Error finding container 2b5645c12b557816f47b8efeccc9ebdb87c644104cac64215adebf545ef9cf89: Status 404 returned error can't find the container with id 2b5645c12b557816f47b8efeccc9ebdb87c644104cac64215adebf545ef9cf89 Jul 20 23:01:48.304530 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.304493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" event={"ID":"019a75c6-e42f-49b1-bcfd-efc5b5f199c3","Type":"ContainerStarted","Data":"2b5645c12b557816f47b8efeccc9ebdb87c644104cac64215adebf545ef9cf89"} Jul 20 23:01:48.502337 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.502300 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-5rjt4"] Jul 20 23:01:48.505806 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.505784 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5rjt4" Jul 20 23:01:48.508786 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.508748 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8qgsb\"" Jul 20 23:01:48.509711 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.509688 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5rjt4"] Jul 20 23:01:48.624416 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.624369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7tqb\" (UniqueName: \"kubernetes.io/projected/7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9-kube-api-access-d7tqb\") pod \"authorino-75d556bd77-5rjt4\" (UID: \"7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9\") " pod="kuadrant-system/authorino-75d556bd77-5rjt4" Jul 20 23:01:48.725023 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.724984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7tqb\" (UniqueName: \"kubernetes.io/projected/7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9-kube-api-access-d7tqb\") pod \"authorino-75d556bd77-5rjt4\" (UID: \"7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9\") " pod="kuadrant-system/authorino-75d556bd77-5rjt4" Jul 20 23:01:48.734723 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.734691 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7tqb\" (UniqueName: \"kubernetes.io/projected/7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9-kube-api-access-d7tqb\") pod \"authorino-75d556bd77-5rjt4\" (UID: \"7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9\") " pod="kuadrant-system/authorino-75d556bd77-5rjt4" Jul 20 23:01:48.818673 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.818636 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5rjt4" Jul 20 23:01:48.969613 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:48.969546 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5rjt4"] Jul 20 23:01:48.972149 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:01:48.972116 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a93ae61_9de2_4ca4_b73b_7cfae63f0ca9.slice/crio-4df8274b415dac6801b32284a95ebe51da94e5e2f619ad6429eea6e60b4358a8 WatchSource:0}: Error finding container 4df8274b415dac6801b32284a95ebe51da94e5e2f619ad6429eea6e60b4358a8: Status 404 returned error can't find the container with id 4df8274b415dac6801b32284a95ebe51da94e5e2f619ad6429eea6e60b4358a8 Jul 20 23:01:49.311117 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:49.311019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5rjt4" event={"ID":"7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9","Type":"ContainerStarted","Data":"4df8274b415dac6801b32284a95ebe51da94e5e2f619ad6429eea6e60b4358a8"} Jul 20 23:01:51.324206 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:51.324120 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" event={"ID":"019a75c6-e42f-49b1-bcfd-efc5b5f199c3","Type":"ContainerStarted","Data":"1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520"} Jul 20 23:01:51.324699 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:51.324248 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:01:51.342985 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:51.342932 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" podStartSLOduration=1.3418526499999999 podStartE2EDuration="4.342915015s" totalImagesPullingTime="3.001062365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:01:51.34153736 +0000 UTC m=+486.659889646" watchObservedRunningTime="2026-07-20 23:01:51.342915015 +0000 UTC m=+486.661267309" Jul 20 23:01:53.335007 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:53.334918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5rjt4" event={"ID":"7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9","Type":"ContainerStarted","Data":"54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e"} Jul 20 23:01:53.351188 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:01:53.351139 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-5rjt4" podStartSLOduration=1.4652973 podStartE2EDuration="5.351123461s" totalImagesPullingTime="3.885826161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:01:53.349753199 +0000 UTC m=+488.668105485" watchObservedRunningTime="2026-07-20 23:01:53.351123461 +0000 UTC m=+488.669475747" Jul 20 23:02:02.331002 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:02.330964 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:02:04.522799 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:04.522762 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5tmtk"] Jul 20 23:02:04.523272 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:04.522943 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" podUID="019a75c6-e42f-49b1-bcfd-efc5b5f199c3" containerName="limitador" containerID="cri-o://1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520" gracePeriod=30 Jul 20 23:02:05.065433 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.065403 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:02:05.176307 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.176271 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bq94\" (UniqueName: \"kubernetes.io/projected/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-kube-api-access-5bq94\") pod \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\" (UID: \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\") " Jul 20 23:02:05.176532 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.176330 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-config-file\") pod \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\" (UID: \"019a75c6-e42f-49b1-bcfd-efc5b5f199c3\") " Jul 20 23:02:05.176791 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.176767 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-config-file" (OuterVolumeSpecName: "config-file") pod "019a75c6-e42f-49b1-bcfd-efc5b5f199c3" (UID: "019a75c6-e42f-49b1-bcfd-efc5b5f199c3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:02:05.178390 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.178361 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-kube-api-access-5bq94" (OuterVolumeSpecName: "kube-api-access-5bq94") pod "019a75c6-e42f-49b1-bcfd-efc5b5f199c3" (UID: "019a75c6-e42f-49b1-bcfd-efc5b5f199c3"). InnerVolumeSpecName "kube-api-access-5bq94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:02:05.277295 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.277256 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bq94\" (UniqueName: \"kubernetes.io/projected/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-kube-api-access-5bq94\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:02:05.277295 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.277291 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/019a75c6-e42f-49b1-bcfd-efc5b5f199c3-config-file\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:02:05.387741 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.387707 2574 generic.go:356] "Generic (PLEG): container finished" podID="019a75c6-e42f-49b1-bcfd-efc5b5f199c3" containerID="1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520" exitCode=0 Jul 20 23:02:05.387902 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.387770 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" Jul 20 23:02:05.387902 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.387791 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" event={"ID":"019a75c6-e42f-49b1-bcfd-efc5b5f199c3","Type":"ContainerDied","Data":"1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520"} Jul 20 23:02:05.387902 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.387848 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5tmtk" event={"ID":"019a75c6-e42f-49b1-bcfd-efc5b5f199c3","Type":"ContainerDied","Data":"2b5645c12b557816f47b8efeccc9ebdb87c644104cac64215adebf545ef9cf89"} Jul 20 23:02:05.387902 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.387874 2574 scope.go:117] "RemoveContainer" containerID="1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520" Jul 20 23:02:05.395731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.395714 2574 scope.go:117] "RemoveContainer" containerID="1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520" Jul 20 23:02:05.395999 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:02:05.395979 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520\": container with ID starting with 1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520 not found: ID does not exist" containerID="1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520" Jul 20 23:02:05.396060 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.396013 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520"} err="failed to get container status \"1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520\": rpc error: code = NotFound desc = could not find container \"1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520\": container with ID starting with 1cc613695b07be05c473b9d03914c2de20a371de0d54bd93841a84a67ab73520 not found: ID does not exist" Jul 20 23:02:05.404584 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.404558 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5tmtk"] Jul 20 23:02:05.408384 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:05.408361 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5tmtk"] Jul 20 23:02:07.227002 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:07.226969 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="019a75c6-e42f-49b1-bcfd-efc5b5f199c3" path="/var/lib/kubelet/pods/019a75c6-e42f-49b1-bcfd-efc5b5f199c3/volumes" Jul 20 23:02:08.919934 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.919899 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-r257d"] Jul 20 23:02:08.920368 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.920351 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019a75c6-e42f-49b1-bcfd-efc5b5f199c3" containerName="limitador" Jul 20 23:02:08.920473 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.920374 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="019a75c6-e42f-49b1-bcfd-efc5b5f199c3" containerName="limitador" Jul 20 23:02:08.920546 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.920498 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="019a75c6-e42f-49b1-bcfd-efc5b5f199c3" containerName="limitador" Jul 20 23:02:08.923578 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.923556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:08.926725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.926702 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 20 23:02:08.926848 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.926751 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 20 23:02:08.926848 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.926702 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:02:08.927874 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.927853 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bmdps\"" Jul 20 23:02:08.931454 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:08.931004 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-r257d"] Jul 20 23:02:09.008813 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:09.008768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d72ea9c8-6b7e-402f-aee8-bf20537bc4fc-data\") pod \"postgres-69fb4594cf-r257d\" (UID: \"d72ea9c8-6b7e-402f-aee8-bf20537bc4fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:09.008813 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:09.008810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc82h\" (UniqueName: \"kubernetes.io/projected/d72ea9c8-6b7e-402f-aee8-bf20537bc4fc-kube-api-access-tc82h\") pod \"postgres-69fb4594cf-r257d\" (UID: \"d72ea9c8-6b7e-402f-aee8-bf20537bc4fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:09.110165 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:09.110122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d72ea9c8-6b7e-402f-aee8-bf20537bc4fc-data\") pod \"postgres-69fb4594cf-r257d\" (UID: \"d72ea9c8-6b7e-402f-aee8-bf20537bc4fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:09.110165 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:09.110162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc82h\" (UniqueName: \"kubernetes.io/projected/d72ea9c8-6b7e-402f-aee8-bf20537bc4fc-kube-api-access-tc82h\") pod \"postgres-69fb4594cf-r257d\" (UID: \"d72ea9c8-6b7e-402f-aee8-bf20537bc4fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:09.110583 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:09.110564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d72ea9c8-6b7e-402f-aee8-bf20537bc4fc-data\") pod \"postgres-69fb4594cf-r257d\" (UID: \"d72ea9c8-6b7e-402f-aee8-bf20537bc4fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:09.118801 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:09.118775 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc82h\" (UniqueName: \"kubernetes.io/projected/d72ea9c8-6b7e-402f-aee8-bf20537bc4fc-kube-api-access-tc82h\") pod \"postgres-69fb4594cf-r257d\" (UID: \"d72ea9c8-6b7e-402f-aee8-bf20537bc4fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:09.235507 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:09.235409 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:09.365761 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:09.365733 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-r257d"] Jul 20 23:02:09.368166 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:02:09.368137 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72ea9c8_6b7e_402f_aee8_bf20537bc4fc.slice/crio-2d35d12c309f410b48c09ecb178c2fb74c6b7a4714cbd59f788746b70a5e743e WatchSource:0}: Error finding container 2d35d12c309f410b48c09ecb178c2fb74c6b7a4714cbd59f788746b70a5e743e: Status 404 returned error can't find the container with id 2d35d12c309f410b48c09ecb178c2fb74c6b7a4714cbd59f788746b70a5e743e Jul 20 23:02:09.406261 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:09.406221 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" event={"ID":"d72ea9c8-6b7e-402f-aee8-bf20537bc4fc","Type":"ContainerStarted","Data":"2d35d12c309f410b48c09ecb178c2fb74c6b7a4714cbd59f788746b70a5e743e"} Jul 20 23:02:14.433066 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:14.433027 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" event={"ID":"d72ea9c8-6b7e-402f-aee8-bf20537bc4fc","Type":"ContainerStarted","Data":"f4a1d5eb0f6e6272f56f505990b217d9f2391286b9ce938230b652c4371f9817"} Jul 20 23:02:14.433654 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:14.433091 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:14.452508 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:14.451488 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" podStartSLOduration=1.495790746 podStartE2EDuration="6.451448383s" totalImagesPullingTime="4.955657637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:02:14.448779291 +0000 UTC m=+509.767131576" watchObservedRunningTime="2026-07-20 23:02:14.451448383 +0000 UTC m=+509.769800669" Jul 20 23:02:20.468058 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:20.468025 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-r257d" Jul 20 23:02:22.666871 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:22.666835 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5rjt4"] Jul 20 23:02:22.667278 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:22.667014 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-5rjt4" podUID="7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9" containerName="authorino" containerID="cri-o://54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e" gracePeriod=30 Jul 20 23:02:22.906801 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:22.906767 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5rjt4" Jul 20 23:02:22.928386 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:22.928309 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7tqb\" (UniqueName: \"kubernetes.io/projected/7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9-kube-api-access-d7tqb\") pod \"7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9\" (UID: \"7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9\") " Jul 20 23:02:22.930428 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:22.930392 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9-kube-api-access-d7tqb" (OuterVolumeSpecName: "kube-api-access-d7tqb") pod "7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9" (UID: "7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9"). InnerVolumeSpecName "kube-api-access-d7tqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:02:23.029670 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.029630 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7tqb\" (UniqueName: \"kubernetes.io/projected/7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9-kube-api-access-d7tqb\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:02:23.472605 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.472566 2574 generic.go:356] "Generic (PLEG): container finished" podID="7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9" containerID="54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e" exitCode=0 Jul 20 23:02:23.472605 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.472609 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5rjt4" Jul 20 23:02:23.472807 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.472626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5rjt4" event={"ID":"7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9","Type":"ContainerDied","Data":"54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e"} Jul 20 23:02:23.472807 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.472666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5rjt4" event={"ID":"7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9","Type":"ContainerDied","Data":"4df8274b415dac6801b32284a95ebe51da94e5e2f619ad6429eea6e60b4358a8"} Jul 20 23:02:23.472807 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.472688 2574 scope.go:117] "RemoveContainer" containerID="54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e" Jul 20 23:02:23.480728 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.480709 2574 scope.go:117] "RemoveContainer" containerID="54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e" Jul 20 23:02:23.480998 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:02:23.480980 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e\": container with ID starting with 54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e not found: ID does not exist" containerID="54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e" Jul 20 23:02:23.481068 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.481006 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e"} err="failed to get container status \"54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e\": rpc error: code = NotFound desc = could not find container \"54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e\": container with ID starting with 54650df5f112f0efb5c34a1755b74b66834dbc8b82e36339606d033809c8c89e not found: ID does not exist" Jul 20 23:02:23.488181 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.488124 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5rjt4"] Jul 20 23:02:23.491619 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:23.491597 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5rjt4"] Jul 20 23:02:25.228358 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:25.228325 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9" path="/var/lib/kubelet/pods/7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9/volumes" Jul 20 23:02:27.165830 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.165787 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-575b7695b5-srn5m"] Jul 20 23:02:27.166225 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.166158 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9" containerName="authorino" Jul 20 23:02:27.166225 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.166170 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9" containerName="authorino" Jul 20 23:02:27.166308 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.166242 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a93ae61-9de2-4ca4-b73b-7cfae63f0ca9" containerName="authorino" Jul 20 23:02:27.174011 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.173982 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:27.177000 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.176976 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 23:02:27.177140 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.177021 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7rshb\"" Jul 20 23:02:27.177140 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.177022 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 23:02:27.182783 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.182753 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-575b7695b5-srn5m"] Jul 20 23:02:27.261871 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.261837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq7g\" (UniqueName: \"kubernetes.io/projected/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-kube-api-access-khq7g\") pod \"maas-controller-575b7695b5-srn5m\" (UID: \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\") " pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:27.262042 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.261909 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-webhook-cert\") pod \"maas-controller-575b7695b5-srn5m\" (UID: \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\") " pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:27.318290 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.318254 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4874f67f-g52hr"] Jul 20 23:02:27.321539 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.321516 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:27.331500 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.331475 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4874f67f-g52hr"] Jul 20 23:02:27.362693 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.362658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaf3b91c-d46c-4142-9138-68ab8fb372ff-webhook-cert\") pod \"maas-controller-c4874f67f-g52hr\" (UID: \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\") " pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:27.362862 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.362701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq7g\" (UniqueName: \"kubernetes.io/projected/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-kube-api-access-khq7g\") pod \"maas-controller-575b7695b5-srn5m\" (UID: \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\") " pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:27.362862 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.362761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7klrf\" (UniqueName: \"kubernetes.io/projected/aaf3b91c-d46c-4142-9138-68ab8fb372ff-kube-api-access-7klrf\") pod \"maas-controller-c4874f67f-g52hr\" (UID: \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\") " pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:27.362862 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.362823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-webhook-cert\") pod \"maas-controller-575b7695b5-srn5m\" (UID: \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\") " pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:27.365238 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.365217 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-webhook-cert\") pod \"maas-controller-575b7695b5-srn5m\" (UID: \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\") " pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:27.372928 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.372899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq7g\" (UniqueName: \"kubernetes.io/projected/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-kube-api-access-khq7g\") pod \"maas-controller-575b7695b5-srn5m\" (UID: \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\") " pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:27.463945 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.463856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaf3b91c-d46c-4142-9138-68ab8fb372ff-webhook-cert\") pod \"maas-controller-c4874f67f-g52hr\" (UID: \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\") " pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:27.463945 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.463907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7klrf\" (UniqueName: \"kubernetes.io/projected/aaf3b91c-d46c-4142-9138-68ab8fb372ff-kube-api-access-7klrf\") pod \"maas-controller-c4874f67f-g52hr\" (UID: \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\") " pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:27.466400 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.466369 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaf3b91c-d46c-4142-9138-68ab8fb372ff-webhook-cert\") pod \"maas-controller-c4874f67f-g52hr\" (UID: \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\") " pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:27.474301 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.474268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7klrf\" (UniqueName: \"kubernetes.io/projected/aaf3b91c-d46c-4142-9138-68ab8fb372ff-kube-api-access-7klrf\") pod \"maas-controller-c4874f67f-g52hr\" (UID: \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\") " pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:27.485133 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.485097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:27.616449 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.616426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-575b7695b5-srn5m"] Jul 20 23:02:27.618229 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:02:27.618201 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5d9b2f_7aa5_46ba_8db1_2014ad6b8d85.slice/crio-f6ddcd0139112c7eceb92ead2b657268d0ee39ccb007dcb30268f534f212b42e WatchSource:0}: Error finding container f6ddcd0139112c7eceb92ead2b657268d0ee39ccb007dcb30268f534f212b42e: Status 404 returned error can't find the container with id f6ddcd0139112c7eceb92ead2b657268d0ee39ccb007dcb30268f534f212b42e Jul 20 23:02:27.631847 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.631824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:27.762596 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:27.762441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4874f67f-g52hr"] Jul 20 23:02:27.764607 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:02:27.764583 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf3b91c_d46c_4142_9138_68ab8fb372ff.slice/crio-987591d70512eab1ccf16d222a33ed937dbdd3eb2d85c407e372336d822e5e0e WatchSource:0}: Error finding container 987591d70512eab1ccf16d222a33ed937dbdd3eb2d85c407e372336d822e5e0e: Status 404 returned error can't find the container with id 987591d70512eab1ccf16d222a33ed937dbdd3eb2d85c407e372336d822e5e0e Jul 20 23:02:28.500236 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:28.500092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575b7695b5-srn5m" event={"ID":"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85","Type":"ContainerStarted","Data":"f6ddcd0139112c7eceb92ead2b657268d0ee39ccb007dcb30268f534f212b42e"} Jul 20 23:02:28.503276 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:28.503186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-g52hr" event={"ID":"aaf3b91c-d46c-4142-9138-68ab8fb372ff","Type":"ContainerStarted","Data":"987591d70512eab1ccf16d222a33ed937dbdd3eb2d85c407e372336d822e5e0e"} Jul 20 23:02:30.514527 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:30.514409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-g52hr" event={"ID":"aaf3b91c-d46c-4142-9138-68ab8fb372ff","Type":"ContainerStarted","Data":"67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e"} Jul 20 23:02:30.514527 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:30.514510 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:30.515871 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:30.515845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575b7695b5-srn5m" event={"ID":"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85","Type":"ContainerStarted","Data":"3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473"} Jul 20 23:02:30.515995 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:30.515893 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:30.548414 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:30.548356 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c4874f67f-g52hr" podStartSLOduration=1.1825287150000001 podStartE2EDuration="3.548338807s" totalImagesPullingTime="2.365810092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:02:30.547253885 +0000 UTC m=+525.865606182" watchObservedRunningTime="2026-07-20 23:02:30.548338807 +0000 UTC m=+525.866691093" Jul 20 23:02:30.581846 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:30.581794 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-575b7695b5-srn5m" podStartSLOduration=1.079173336 podStartE2EDuration="3.581778727s" totalImagesPullingTime="2.502605391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:02:30.579721073 +0000 UTC m=+525.898073359" watchObservedRunningTime="2026-07-20 23:02:30.581778727 +0000 UTC m=+525.900131037" Jul 20 23:02:41.522931 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.522844 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:02:41.522931 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.522921 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:41.569228 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.569195 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-575b7695b5-srn5m"] Jul 20 23:02:41.569409 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.569368 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-575b7695b5-srn5m" podUID="7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85" containerName="manager" containerID="cri-o://3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473" gracePeriod=30 Jul 20 23:02:41.818748 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.818720 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:41.880726 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.880689 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-webhook-cert\") pod \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\" (UID: \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\") " Jul 20 23:02:41.880887 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.880751 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khq7g\" (UniqueName: \"kubernetes.io/projected/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-kube-api-access-khq7g\") pod \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\" (UID: \"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85\") " Jul 20 23:02:41.882933 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.882881 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85" (UID: "7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:02:41.883048 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.882950 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-kube-api-access-khq7g" (OuterVolumeSpecName: "kube-api-access-khq7g") pod "7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85" (UID: "7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85"). InnerVolumeSpecName "kube-api-access-khq7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:02:41.982093 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.982051 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khq7g\" (UniqueName: \"kubernetes.io/projected/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-kube-api-access-khq7g\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:02:41.982093 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:41.982083 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85-webhook-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:02:42.569634 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:42.569597 2574 generic.go:356] "Generic (PLEG): container finished" podID="7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85" containerID="3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473" exitCode=0 Jul 20 23:02:42.570077 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:42.569641 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-575b7695b5-srn5m" Jul 20 23:02:42.570077 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:42.569640 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575b7695b5-srn5m" event={"ID":"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85","Type":"ContainerDied","Data":"3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473"} Jul 20 23:02:42.570077 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:42.569757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-575b7695b5-srn5m" event={"ID":"7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85","Type":"ContainerDied","Data":"f6ddcd0139112c7eceb92ead2b657268d0ee39ccb007dcb30268f534f212b42e"} Jul 20 23:02:42.570077 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:42.569787 2574 scope.go:117] "RemoveContainer" containerID="3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473" Jul 20 23:02:42.577920 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:42.577901 2574 scope.go:117] "RemoveContainer" containerID="3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473" Jul 20 23:02:42.578330 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:02:42.578293 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473\": container with ID starting with 3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473 not found: ID does not exist" containerID="3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473" Jul 20 23:02:42.578438 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:42.578338 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473"} err="failed to get container status \"3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473\": rpc error: code = NotFound desc = could not find container \"3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473\": container with ID starting with 3a144e45bcb3df5ed2193c6acfff8e8fd00f585790f5ec2652ce343786801473 not found: ID does not exist" Jul 20 23:02:42.592959 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:42.592931 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-575b7695b5-srn5m"] Jul 20 23:02:42.594669 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:42.594649 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-575b7695b5-srn5m"] Jul 20 23:02:43.227679 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:02:43.227647 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85" path="/var/lib/kubelet/pods/7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85/volumes" Jul 20 23:03:02.719419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.719382 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz"] Jul 20 23:03:02.719843 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.719709 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85" containerName="manager" Jul 20 23:03:02.719843 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.719722 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85" containerName="manager" Jul 20 23:03:02.719843 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.719787 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d5d9b2f-7aa5-46ba-8db1-2014ad6b8d85" containerName="manager" Jul 20 23:03:02.746493 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.746449 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz"] Jul 20 23:03:02.746632 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.746574 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:02.749794 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.749769 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fqkqk\"" Jul 20 23:03:02.750031 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.750014 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 20 23:03:02.858867 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.858831 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d276f814-e5c0-4612-9104-a316a8c870d6-maas-api-tls\") pod \"maas-api-5f68cf4fbd-pj6bz\" (UID: \"d276f814-e5c0-4612-9104-a316a8c870d6\") " pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:02.858867 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.858873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbz4n\" (UniqueName: \"kubernetes.io/projected/d276f814-e5c0-4612-9104-a316a8c870d6-kube-api-access-sbz4n\") pod \"maas-api-5f68cf4fbd-pj6bz\" (UID: \"d276f814-e5c0-4612-9104-a316a8c870d6\") " pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:02.859080 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.858903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d276f814-e5c0-4612-9104-a316a8c870d6-openshift-service-ca\") pod \"maas-api-5f68cf4fbd-pj6bz\" (UID: \"d276f814-e5c0-4612-9104-a316a8c870d6\") " pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:02.959418 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.959379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d276f814-e5c0-4612-9104-a316a8c870d6-maas-api-tls\") pod \"maas-api-5f68cf4fbd-pj6bz\" (UID: \"d276f814-e5c0-4612-9104-a316a8c870d6\") " pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:02.959418 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.959421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbz4n\" (UniqueName: \"kubernetes.io/projected/d276f814-e5c0-4612-9104-a316a8c870d6-kube-api-access-sbz4n\") pod \"maas-api-5f68cf4fbd-pj6bz\" (UID: \"d276f814-e5c0-4612-9104-a316a8c870d6\") " pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:02.959661 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.959502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d276f814-e5c0-4612-9104-a316a8c870d6-openshift-service-ca\") pod \"maas-api-5f68cf4fbd-pj6bz\" (UID: \"d276f814-e5c0-4612-9104-a316a8c870d6\") " pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:02.960021 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.959996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d276f814-e5c0-4612-9104-a316a8c870d6-openshift-service-ca\") pod \"maas-api-5f68cf4fbd-pj6bz\" (UID: \"d276f814-e5c0-4612-9104-a316a8c870d6\") " pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:02.961961 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.961941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d276f814-e5c0-4612-9104-a316a8c870d6-maas-api-tls\") pod \"maas-api-5f68cf4fbd-pj6bz\" (UID: \"d276f814-e5c0-4612-9104-a316a8c870d6\") " pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:02.972880 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:02.972816 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbz4n\" (UniqueName: \"kubernetes.io/projected/d276f814-e5c0-4612-9104-a316a8c870d6-kube-api-access-sbz4n\") pod \"maas-api-5f68cf4fbd-pj6bz\" (UID: \"d276f814-e5c0-4612-9104-a316a8c870d6\") " pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:03.057164 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.057125 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:03.199334 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.199236 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz"] Jul 20 23:03:03.202694 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:03:03.202659 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd276f814_e5c0_4612_9104_a316a8c870d6.slice/crio-446f95a3b9aad3b705c9ca7b40212b399ccb10480d53ecf23145aff0d617725a WatchSource:0}: Error finding container 446f95a3b9aad3b705c9ca7b40212b399ccb10480d53ecf23145aff0d617725a: Status 404 returned error can't find the container with id 446f95a3b9aad3b705c9ca7b40212b399ccb10480d53ecf23145aff0d617725a Jul 20 23:03:03.336994 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.336912 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-mt455"] Jul 20 23:03:03.341503 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.341485 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:03.344490 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.344442 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 20 23:03:03.344622 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.344441 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rxhv7\"" Jul 20 23:03:03.351414 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.351388 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-mt455"] Jul 20 23:03:03.462994 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.462949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc0e005-0b90-4d48-9213-9d8ece94a5b7-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-mt455\" (UID: \"5fc0e005-0b90-4d48-9213-9d8ece94a5b7\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:03.463197 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.463024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmll\" (UniqueName: \"kubernetes.io/projected/5fc0e005-0b90-4d48-9213-9d8ece94a5b7-kube-api-access-bjmll\") pod \"payload-pre-processing-7677b774bb-mt455\" (UID: \"5fc0e005-0b90-4d48-9213-9d8ece94a5b7\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:03.564316 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.564266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmll\" (UniqueName: \"kubernetes.io/projected/5fc0e005-0b90-4d48-9213-9d8ece94a5b7-kube-api-access-bjmll\") pod \"payload-pre-processing-7677b774bb-mt455\" (UID: \"5fc0e005-0b90-4d48-9213-9d8ece94a5b7\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:03.564548 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.564372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc0e005-0b90-4d48-9213-9d8ece94a5b7-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-mt455\" (UID: \"5fc0e005-0b90-4d48-9213-9d8ece94a5b7\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:03.565115 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.565092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc0e005-0b90-4d48-9213-9d8ece94a5b7-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-mt455\" (UID: \"5fc0e005-0b90-4d48-9213-9d8ece94a5b7\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:03.573989 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.573957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmll\" (UniqueName: \"kubernetes.io/projected/5fc0e005-0b90-4d48-9213-9d8ece94a5b7-kube-api-access-bjmll\") pod \"payload-pre-processing-7677b774bb-mt455\" (UID: \"5fc0e005-0b90-4d48-9213-9d8ece94a5b7\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:03.652808 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.652768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:03.661848 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.661815 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" event={"ID":"d276f814-e5c0-4612-9104-a316a8c870d6","Type":"ContainerStarted","Data":"446f95a3b9aad3b705c9ca7b40212b399ccb10480d53ecf23145aff0d617725a"} Jul 20 23:03:03.791187 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:03.791156 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-mt455"] Jul 20 23:03:03.793618 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:03:03.793586 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc0e005_0b90_4d48_9213_9d8ece94a5b7.slice/crio-34a57d25458ae1a8a40b53d51ec9095b3b24b7de98b835ca674733adae51b990 WatchSource:0}: Error finding container 34a57d25458ae1a8a40b53d51ec9095b3b24b7de98b835ca674733adae51b990: Status 404 returned error can't find the container with id 34a57d25458ae1a8a40b53d51ec9095b3b24b7de98b835ca674733adae51b990 Jul 20 23:03:04.691745 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:04.691713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" event={"ID":"5fc0e005-0b90-4d48-9213-9d8ece94a5b7","Type":"ContainerStarted","Data":"34a57d25458ae1a8a40b53d51ec9095b3b24b7de98b835ca674733adae51b990"} Jul 20 23:03:05.697953 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:05.697910 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" event={"ID":"d276f814-e5c0-4612-9104-a316a8c870d6","Type":"ContainerStarted","Data":"76680ef3145f91e8696b69a242d9832d6cc8e105e5456ef2f0f19d020a086cd3"} Jul 20 23:03:05.698360 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:05.697979 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:05.722334 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:05.719189 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" podStartSLOduration=1.7903050409999999 podStartE2EDuration="3.719168046s" totalImagesPullingTime="1.928863005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:03:05.715312623 +0000 UTC m=+561.033664911" watchObservedRunningTime="2026-07-20 23:03:05.719168046 +0000 UTC m=+561.037520333" Jul 20 23:03:07.708782 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:07.708751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" event={"ID":"5fc0e005-0b90-4d48-9213-9d8ece94a5b7","Type":"ContainerStarted","Data":"ad648798f391fab3df0349e380d0c9b5c363b299ec896cc129f7eeda550a57f4"} Jul 20 23:03:07.709214 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:07.708894 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:07.726131 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:07.726076 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" podStartSLOduration=1.721119006 podStartE2EDuration="4.726058642s" totalImagesPullingTime="3.004939636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:03:07.725054417 +0000 UTC m=+563.043406703" watchObservedRunningTime="2026-07-20 23:03:07.726058642 +0000 UTC m=+563.044410971" Jul 20 23:03:11.708832 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:11.708798 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f68cf4fbd-pj6bz" Jul 20 23:03:11.730043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:11.730013 2574 generic.go:356] "Generic (PLEG): container finished" podID="5bf0978f-511a-4503-a7c0-8321d105b86f" containerID="56e3c57886e97aec262294acae6a4eb16c47640a8af394a9c2774b6b2cf28d62" exitCode=0 Jul 20 23:03:11.730204 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:11.730083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" event={"ID":"5bf0978f-511a-4503-a7c0-8321d105b86f","Type":"ContainerDied","Data":"56e3c57886e97aec262294acae6a4eb16c47640a8af394a9c2774b6b2cf28d62"} Jul 20 23:03:11.730516 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:11.730497 2574 scope.go:117] "RemoveContainer" containerID="56e3c57886e97aec262294acae6a4eb16c47640a8af394a9c2774b6b2cf28d62" Jul 20 23:03:11.731449 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:11.731423 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:03:11.731558 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:11.731504 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:03:11.731558 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:11.731536 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:03:12.736041 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:12.735998 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" event={"ID":"5bf0978f-511a-4503-a7c0-8321d105b86f","Type":"ContainerStarted","Data":"e1ba811053fa8bb3834b56a2603c8ee63c6adf4ae7ebfe82e37dfd4ed78514bc"} Jul 20 23:03:13.277914 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:13.277879 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:03:13.278924 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:13.278896 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-49zl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 20 23:03:13.279139 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:13.279115 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" podUID="5bf0978f-511a-4503-a7c0-8321d105b86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 20 23:03:14.278519 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:14.278480 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-49zl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 20 23:03:14.279027 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:14.278551 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" podUID="5bf0978f-511a-4503-a7c0-8321d105b86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 20 23:03:15.277493 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:15.277433 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-49zl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 20 23:03:15.277696 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:15.277530 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" podUID="5bf0978f-511a-4503-a7c0-8321d105b86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 20 23:03:16.278375 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:16.278331 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-49zl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 20 23:03:16.278971 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:16.278938 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" podUID="5bf0978f-511a-4503-a7c0-8321d105b86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 20 23:03:17.281690 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:17.281648 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:03:17.282053 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:17.281946 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:03:17.282521 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:17.282501 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-49zl4" Jul 20 23:03:18.717210 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:18.717169 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-mt455" Jul 20 23:03:28.615415 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.615380 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj"] Jul 20 23:03:28.620417 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.620393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.623367 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.623342 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 23:03:28.624675 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.624653 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tmj2g\"" Jul 20 23:03:28.624784 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.624653 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 20 23:03:28.624784 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.624728 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 23:03:28.630671 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.630646 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj"] Jul 20 23:03:28.700647 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.700615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.700839 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.700721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.700839 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.700744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.700839 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.700800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.700839 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.700820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.701014 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.700852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfwmk\" (UniqueName: \"kubernetes.io/projected/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-kube-api-access-gfwmk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.802304 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.802271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.802488 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.802343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.802488 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.802362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.802488 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.802391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.802488 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.802406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.802488 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.802428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfwmk\" (UniqueName: \"kubernetes.io/projected/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-kube-api-access-gfwmk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.802833 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.802807 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.802902 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.802874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.802954 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.802905 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.804652 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.804626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.804878 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.804859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.810733 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.810711 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfwmk\" (UniqueName: \"kubernetes.io/projected/cc38ab44-9d53-4c4c-bc95-f2d19a0818eb-kube-api-access-gfwmk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj\" (UID: \"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:28.933562 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:28.933434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:29.068485 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:29.068443 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj"] Jul 20 23:03:29.070638 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:03:29.070611 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc38ab44_9d53_4c4c_bc95_f2d19a0818eb.slice/crio-7aa54e12bd13e54f880d725471b029e16a26a0a82a770abcc19f8178ec26d719 WatchSource:0}: Error finding container 7aa54e12bd13e54f880d725471b029e16a26a0a82a770abcc19f8178ec26d719: Status 404 returned error can't find the container with id 7aa54e12bd13e54f880d725471b029e16a26a0a82a770abcc19f8178ec26d719 Jul 20 23:03:29.812865 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:29.812826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" event={"ID":"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb","Type":"ContainerStarted","Data":"7aa54e12bd13e54f880d725471b029e16a26a0a82a770abcc19f8178ec26d719"} Jul 20 23:03:30.818151 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:30.818115 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" event={"ID":"cc38ab44-9d53-4c4c-bc95-f2d19a0818eb","Type":"ContainerStarted","Data":"51b7af3c0a9a3a3acc2dab12d8505392d1d6121536b193d8c2685faec48b0805"} Jul 20 23:03:30.836532 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:30.836482 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" podStartSLOduration=1.7177533230000002 podStartE2EDuration="2.836448836s" totalImagesPullingTime="1.118695513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:03:30.835747959 +0000 UTC m=+586.154100245" watchObservedRunningTime="2026-07-20 23:03:30.836448836 +0000 UTC m=+586.154801160" Jul 20 23:03:38.934483 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:38.934411 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:38.934966 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:38.934509 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:38.947048 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:38.947024 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:39.870807 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:39.870775 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj" Jul 20 23:03:41.113178 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.113141 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx"] Jul 20 23:03:41.116857 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.116826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.119529 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.119506 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 20 23:03:41.129372 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.129347 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx"] Jul 20 23:03:41.205723 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.205688 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.205893 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.205732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.205893 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.205797 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.205893 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.205825 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1001a1c9-445c-4e3b-9605-ae5f3d952f35-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.205893 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.205875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd6zt\" (UniqueName: \"kubernetes.io/projected/1001a1c9-445c-4e3b-9605-ae5f3d952f35-kube-api-access-rd6zt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.206032 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.205898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.306797 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.306749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.307003 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.306813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.307003 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.306844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1001a1c9-445c-4e3b-9605-ae5f3d952f35-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.307003 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.306888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd6zt\" (UniqueName: \"kubernetes.io/projected/1001a1c9-445c-4e3b-9605-ae5f3d952f35-kube-api-access-rd6zt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.307163 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.307014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.307163 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.307089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.307251 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.307216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.307357 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.307336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.307412 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.307382 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.309140 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.309117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1001a1c9-445c-4e3b-9605-ae5f3d952f35-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.309393 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.309375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1001a1c9-445c-4e3b-9605-ae5f3d952f35-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.315216 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.315190 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd6zt\" (UniqueName: \"kubernetes.io/projected/1001a1c9-445c-4e3b-9605-ae5f3d952f35-kube-api-access-rd6zt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zq5gx\" (UID: \"1001a1c9-445c-4e3b-9605-ae5f3d952f35\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.428311 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.428205 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:41.562045 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.562016 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx"] Jul 20 23:03:41.563726 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:03:41.563698 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1001a1c9_445c_4e3b_9605_ae5f3d952f35.slice/crio-10cbce6ce2452013f701094a92e8d9d2f795614fe6563de84d6a556776d5f485 WatchSource:0}: Error finding container 10cbce6ce2452013f701094a92e8d9d2f795614fe6563de84d6a556776d5f485: Status 404 returned error can't find the container with id 10cbce6ce2452013f701094a92e8d9d2f795614fe6563de84d6a556776d5f485 Jul 20 23:03:41.873180 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.873144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" event={"ID":"1001a1c9-445c-4e3b-9605-ae5f3d952f35","Type":"ContainerStarted","Data":"915f1a0125fd7ac6f3bbec41676b3d7f16fae30ce64f4ceafe53972b2595abd6"} Jul 20 23:03:41.873357 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.873196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" event={"ID":"1001a1c9-445c-4e3b-9605-ae5f3d952f35","Type":"ContainerStarted","Data":"10cbce6ce2452013f701094a92e8d9d2f795614fe6563de84d6a556776d5f485"} Jul 20 23:03:41.891377 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:41.891318 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" podStartSLOduration=0.713219795 podStartE2EDuration="891.302978ms" totalImagesPullingTime="178.083183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:03:41.8894423 +0000 UTC m=+597.207794597" watchObservedRunningTime="2026-07-20 23:03:41.891302978 +0000 UTC m=+597.209655263" Jul 20 23:03:45.071104 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:45.071072 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:03:45.072878 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:45.072853 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:03:51.428357 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:51.428315 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:51.428813 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:51.428384 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:51.441018 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:51.440987 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:03:51.933094 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:03:51.933065 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zq5gx" Jul 20 23:04:53.028063 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.028023 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-769d7847f9-kbhrz"] Jul 20 23:04:53.031411 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.031393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769d7847f9-kbhrz" Jul 20 23:04:53.032646 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.032625 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nqf8\" (UniqueName: \"kubernetes.io/projected/6c141d47-8a38-428d-bb1d-e77acd66cb92-kube-api-access-9nqf8\") pod \"authorino-769d7847f9-kbhrz\" (UID: \"6c141d47-8a38-428d-bb1d-e77acd66cb92\") " pod="kuadrant-system/authorino-769d7847f9-kbhrz" Jul 20 23:04:53.032721 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.032703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c141d47-8a38-428d-bb1d-e77acd66cb92-tls-cert\") pod \"authorino-769d7847f9-kbhrz\" (UID: \"6c141d47-8a38-428d-bb1d-e77acd66cb92\") " pod="kuadrant-system/authorino-769d7847f9-kbhrz" Jul 20 23:04:53.035627 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.035603 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 20 23:04:53.035717 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.035603 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8qgsb\"" Jul 20 23:04:53.038935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.038909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-769d7847f9-kbhrz"] Jul 20 23:04:53.133764 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.133713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqf8\" (UniqueName: \"kubernetes.io/projected/6c141d47-8a38-428d-bb1d-e77acd66cb92-kube-api-access-9nqf8\") pod \"authorino-769d7847f9-kbhrz\" (UID: \"6c141d47-8a38-428d-bb1d-e77acd66cb92\") " pod="kuadrant-system/authorino-769d7847f9-kbhrz" Jul 20 23:04:53.133944 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.133830 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c141d47-8a38-428d-bb1d-e77acd66cb92-tls-cert\") pod \"authorino-769d7847f9-kbhrz\" (UID: \"6c141d47-8a38-428d-bb1d-e77acd66cb92\") " pod="kuadrant-system/authorino-769d7847f9-kbhrz" Jul 20 23:04:53.136316 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.136288 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6c141d47-8a38-428d-bb1d-e77acd66cb92-tls-cert\") pod \"authorino-769d7847f9-kbhrz\" (UID: \"6c141d47-8a38-428d-bb1d-e77acd66cb92\") " pod="kuadrant-system/authorino-769d7847f9-kbhrz" Jul 20 23:04:53.142749 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.142726 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqf8\" (UniqueName: \"kubernetes.io/projected/6c141d47-8a38-428d-bb1d-e77acd66cb92-kube-api-access-9nqf8\") pod \"authorino-769d7847f9-kbhrz\" (UID: \"6c141d47-8a38-428d-bb1d-e77acd66cb92\") " pod="kuadrant-system/authorino-769d7847f9-kbhrz" Jul 20 23:04:53.341965 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.341870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769d7847f9-kbhrz" Jul 20 23:04:53.489283 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.489250 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-769d7847f9-kbhrz"] Jul 20 23:04:53.492691 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:04:53.492662 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c141d47_8a38_428d_bb1d_e77acd66cb92.slice/crio-819afd80c05a3ecc01625f097dbcd9dbe7150eec21ad70a3708f841f23f4038c WatchSource:0}: Error finding container 819afd80c05a3ecc01625f097dbcd9dbe7150eec21ad70a3708f841f23f4038c: Status 404 returned error can't find the container with id 819afd80c05a3ecc01625f097dbcd9dbe7150eec21ad70a3708f841f23f4038c Jul 20 23:04:53.494313 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:53.494295 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 23:04:54.206088 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:54.205992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769d7847f9-kbhrz" event={"ID":"6c141d47-8a38-428d-bb1d-e77acd66cb92","Type":"ContainerStarted","Data":"69d19273241d8ac88fa2a80d1ea49337a10caf6e0bfdce7e6b377ffd2cbd6d16"} Jul 20 23:04:54.206088 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:54.206047 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769d7847f9-kbhrz" event={"ID":"6c141d47-8a38-428d-bb1d-e77acd66cb92","Type":"ContainerStarted","Data":"819afd80c05a3ecc01625f097dbcd9dbe7150eec21ad70a3708f841f23f4038c"} Jul 20 23:04:54.233093 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:04:54.233035 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-769d7847f9-kbhrz" podStartSLOduration=0.815355825 podStartE2EDuration="1.233016354s" totalImagesPullingTime="417.660529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:04:54.232652003 +0000 UTC m=+669.551004303" watchObservedRunningTime="2026-07-20 23:04:54.233016354 +0000 UTC m=+669.551368641" Jul 20 23:05:39.547914 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:39.547833 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4874f67f-g52hr"] Jul 20 23:05:39.548389 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:39.548030 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c4874f67f-g52hr" podUID="aaf3b91c-d46c-4142-9138-68ab8fb372ff" containerName="manager" containerID="cri-o://67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e" gracePeriod=30 Jul 20 23:05:39.803671 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:39.803585 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:05:39.856664 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:39.856633 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaf3b91c-d46c-4142-9138-68ab8fb372ff-webhook-cert\") pod \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\" (UID: \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\") " Jul 20 23:05:39.856830 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:39.856749 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7klrf\" (UniqueName: \"kubernetes.io/projected/aaf3b91c-d46c-4142-9138-68ab8fb372ff-kube-api-access-7klrf\") pod \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\" (UID: \"aaf3b91c-d46c-4142-9138-68ab8fb372ff\") " Jul 20 23:05:39.859000 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:39.858838 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf3b91c-d46c-4142-9138-68ab8fb372ff-kube-api-access-7klrf" (OuterVolumeSpecName: "kube-api-access-7klrf") pod "aaf3b91c-d46c-4142-9138-68ab8fb372ff" (UID: "aaf3b91c-d46c-4142-9138-68ab8fb372ff"). InnerVolumeSpecName "kube-api-access-7klrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:05:39.859286 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:39.859258 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf3b91c-d46c-4142-9138-68ab8fb372ff-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "aaf3b91c-d46c-4142-9138-68ab8fb372ff" (UID: "aaf3b91c-d46c-4142-9138-68ab8fb372ff"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:05:39.957862 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:39.957825 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aaf3b91c-d46c-4142-9138-68ab8fb372ff-webhook-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:05:39.957862 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:39.957857 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7klrf\" (UniqueName: \"kubernetes.io/projected/aaf3b91c-d46c-4142-9138-68ab8fb372ff-kube-api-access-7klrf\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:05:40.419764 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:40.419730 2574 generic.go:356] "Generic (PLEG): container finished" podID="aaf3b91c-d46c-4142-9138-68ab8fb372ff" containerID="67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e" exitCode=0 Jul 20 23:05:40.419940 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:40.419783 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-g52hr" event={"ID":"aaf3b91c-d46c-4142-9138-68ab8fb372ff","Type":"ContainerDied","Data":"67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e"} Jul 20 23:05:40.419940 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:40.419795 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-g52hr" Jul 20 23:05:40.419940 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:40.419825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-g52hr" event={"ID":"aaf3b91c-d46c-4142-9138-68ab8fb372ff","Type":"ContainerDied","Data":"987591d70512eab1ccf16d222a33ed937dbdd3eb2d85c407e372336d822e5e0e"} Jul 20 23:05:40.419940 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:40.419847 2574 scope.go:117] "RemoveContainer" containerID="67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e" Jul 20 23:05:40.429001 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:40.428975 2574 scope.go:117] "RemoveContainer" containerID="67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e" Jul 20 23:05:40.429256 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:05:40.429236 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e\": container with ID starting with 67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e not found: ID does not exist" containerID="67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e" Jul 20 23:05:40.429305 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:40.429266 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e"} err="failed to get container status \"67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e\": rpc error: code = NotFound desc = could not find container \"67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e\": container with ID starting with 67a87fd0e940af41dba3971407bdc5cac8186ff58ee6a7d778197b66d52c9a9e not found: ID does not exist" Jul 20 23:05:40.441639 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:40.441614 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4874f67f-g52hr"] Jul 20 23:05:40.447306 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:40.447277 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c4874f67f-g52hr"] Jul 20 23:05:41.228852 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:05:41.228821 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaf3b91c-d46c-4142-9138-68ab8fb372ff" path="/var/lib/kubelet/pods/aaf3b91c-d46c-4142-9138-68ab8fb372ff/volumes" Jul 20 23:06:41.217832 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.217791 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4874f67f-kbkfg"] Jul 20 23:06:41.218342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.218277 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf3b91c-d46c-4142-9138-68ab8fb372ff" containerName="manager" Jul 20 23:06:41.218342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.218297 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf3b91c-d46c-4142-9138-68ab8fb372ff" containerName="manager" Jul 20 23:06:41.218496 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.218418 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaf3b91c-d46c-4142-9138-68ab8fb372ff" containerName="manager" Jul 20 23:06:41.221313 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.221291 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:41.226391 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.226215 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 23:06:41.226391 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.226260 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7rshb\"" Jul 20 23:06:41.226592 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.226404 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 23:06:41.232959 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.232937 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4874f67f-kbkfg"] Jul 20 23:06:41.286667 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.286630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f298b325-df8d-4d2d-999e-98330058f94c-webhook-cert\") pod \"maas-controller-c4874f67f-kbkfg\" (UID: \"f298b325-df8d-4d2d-999e-98330058f94c\") " pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:41.286853 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.286729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5kg\" (UniqueName: \"kubernetes.io/projected/f298b325-df8d-4d2d-999e-98330058f94c-kube-api-access-wp5kg\") pod \"maas-controller-c4874f67f-kbkfg\" (UID: \"f298b325-df8d-4d2d-999e-98330058f94c\") " pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:41.387990 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.387949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5kg\" (UniqueName: \"kubernetes.io/projected/f298b325-df8d-4d2d-999e-98330058f94c-kube-api-access-wp5kg\") pod \"maas-controller-c4874f67f-kbkfg\" (UID: \"f298b325-df8d-4d2d-999e-98330058f94c\") " pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:41.388153 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.388008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f298b325-df8d-4d2d-999e-98330058f94c-webhook-cert\") pod \"maas-controller-c4874f67f-kbkfg\" (UID: \"f298b325-df8d-4d2d-999e-98330058f94c\") " pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:41.390379 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.390354 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f298b325-df8d-4d2d-999e-98330058f94c-webhook-cert\") pod \"maas-controller-c4874f67f-kbkfg\" (UID: \"f298b325-df8d-4d2d-999e-98330058f94c\") " pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:41.397894 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.397868 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5kg\" (UniqueName: \"kubernetes.io/projected/f298b325-df8d-4d2d-999e-98330058f94c-kube-api-access-wp5kg\") pod \"maas-controller-c4874f67f-kbkfg\" (UID: \"f298b325-df8d-4d2d-999e-98330058f94c\") " pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:41.537081 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.536986 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:41.667998 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.667945 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4874f67f-kbkfg"] Jul 20 23:06:41.670238 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:06:41.670200 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf298b325_df8d_4d2d_999e_98330058f94c.slice/crio-c01be220617e422ef586833a797888d3a18484db24bea15495a27869279d7de0 WatchSource:0}: Error finding container c01be220617e422ef586833a797888d3a18484db24bea15495a27869279d7de0: Status 404 returned error can't find the container with id c01be220617e422ef586833a797888d3a18484db24bea15495a27869279d7de0 Jul 20 23:06:41.705207 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:41.705169 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-kbkfg" event={"ID":"f298b325-df8d-4d2d-999e-98330058f94c","Type":"ContainerStarted","Data":"c01be220617e422ef586833a797888d3a18484db24bea15495a27869279d7de0"} Jul 20 23:06:42.711078 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:42.711043 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-kbkfg" event={"ID":"f298b325-df8d-4d2d-999e-98330058f94c","Type":"ContainerStarted","Data":"6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0"} Jul 20 23:06:42.711479 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:42.711110 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:42.729735 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:42.729681 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c4874f67f-kbkfg" podStartSLOduration=1.151839352 podStartE2EDuration="1.72966165s" totalImagesPullingTime="577.822298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:06:42.729379116 +0000 UTC m=+778.047731403" watchObservedRunningTime="2026-07-20 23:06:42.72966165 +0000 UTC m=+778.048013937" Jul 20 23:06:53.717725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:53.717688 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:58.346228 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.346191 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4874f67f-kbkfg"] Jul 20 23:06:58.346769 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.346405 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c4874f67f-kbkfg" podUID="f298b325-df8d-4d2d-999e-98330058f94c" containerName="manager" containerID="cri-o://6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0" gracePeriod=30 Jul 20 23:06:58.588278 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.588251 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:58.740946 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.740909 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f298b325-df8d-4d2d-999e-98330058f94c-webhook-cert\") pod \"f298b325-df8d-4d2d-999e-98330058f94c\" (UID: \"f298b325-df8d-4d2d-999e-98330058f94c\") " Jul 20 23:06:58.741124 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.741009 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp5kg\" (UniqueName: \"kubernetes.io/projected/f298b325-df8d-4d2d-999e-98330058f94c-kube-api-access-wp5kg\") pod \"f298b325-df8d-4d2d-999e-98330058f94c\" (UID: \"f298b325-df8d-4d2d-999e-98330058f94c\") " Jul 20 23:06:58.743088 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.743053 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f298b325-df8d-4d2d-999e-98330058f94c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f298b325-df8d-4d2d-999e-98330058f94c" (UID: "f298b325-df8d-4d2d-999e-98330058f94c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:06:58.743210 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.743148 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f298b325-df8d-4d2d-999e-98330058f94c-kube-api-access-wp5kg" (OuterVolumeSpecName: "kube-api-access-wp5kg") pod "f298b325-df8d-4d2d-999e-98330058f94c" (UID: "f298b325-df8d-4d2d-999e-98330058f94c"). InnerVolumeSpecName "kube-api-access-wp5kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:06:58.780300 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.780270 2574 generic.go:356] "Generic (PLEG): container finished" podID="f298b325-df8d-4d2d-999e-98330058f94c" containerID="6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0" exitCode=0 Jul 20 23:06:58.780511 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.780315 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-kbkfg" Jul 20 23:06:58.780511 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.780351 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-kbkfg" event={"ID":"f298b325-df8d-4d2d-999e-98330058f94c","Type":"ContainerDied","Data":"6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0"} Jul 20 23:06:58.780511 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.780402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-kbkfg" event={"ID":"f298b325-df8d-4d2d-999e-98330058f94c","Type":"ContainerDied","Data":"c01be220617e422ef586833a797888d3a18484db24bea15495a27869279d7de0"} Jul 20 23:06:58.780511 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.780423 2574 scope.go:117] "RemoveContainer" containerID="6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0" Jul 20 23:06:58.789067 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.789052 2574 scope.go:117] "RemoveContainer" containerID="6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0" Jul 20 23:06:58.789329 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:06:58.789311 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0\": container with ID starting with 6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0 not found: ID does not exist" containerID="6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0" Jul 20 23:06:58.789366 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.789337 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0"} err="failed to get container status \"6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0\": rpc error: code = NotFound desc = could not find container \"6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0\": container with ID starting with 6d823f7e2bebf5c46c8334011eebb0c6b99ddd005f912f632b78798b6a300ea0 not found: ID does not exist" Jul 20 23:06:58.802247 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.802216 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4874f67f-kbkfg"] Jul 20 23:06:58.807603 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.807578 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c4874f67f-kbkfg"] Jul 20 23:06:58.841837 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.841815 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp5kg\" (UniqueName: \"kubernetes.io/projected/f298b325-df8d-4d2d-999e-98330058f94c-kube-api-access-wp5kg\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:06:58.841837 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:58.841837 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f298b325-df8d-4d2d-999e-98330058f94c-webhook-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:06:59.227335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.227291 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f298b325-df8d-4d2d-999e-98330058f94c" path="/var/lib/kubelet/pods/f298b325-df8d-4d2d-999e-98330058f94c/volumes" Jul 20 23:06:59.563989 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.563910 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4874f67f-pckkq"] Jul 20 23:06:59.564335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.564243 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f298b325-df8d-4d2d-999e-98330058f94c" containerName="manager" Jul 20 23:06:59.564335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.564253 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f298b325-df8d-4d2d-999e-98330058f94c" containerName="manager" Jul 20 23:06:59.564335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.564318 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f298b325-df8d-4d2d-999e-98330058f94c" containerName="manager" Jul 20 23:06:59.568524 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.568506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:06:59.572486 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.572449 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 23:06:59.572611 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.572484 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 23:06:59.572611 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.572449 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7rshb\"" Jul 20 23:06:59.578175 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.578155 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4874f67f-pckkq"] Jul 20 23:06:59.646915 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.646876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppp9z\" (UniqueName: \"kubernetes.io/projected/f3d819a3-29f2-4992-b070-c85423fa6d1e-kube-api-access-ppp9z\") pod \"maas-controller-c4874f67f-pckkq\" (UID: \"f3d819a3-29f2-4992-b070-c85423fa6d1e\") " pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:06:59.647081 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.646933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3d819a3-29f2-4992-b070-c85423fa6d1e-webhook-cert\") pod \"maas-controller-c4874f67f-pckkq\" (UID: \"f3d819a3-29f2-4992-b070-c85423fa6d1e\") " pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:06:59.747593 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.747561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp9z\" (UniqueName: \"kubernetes.io/projected/f3d819a3-29f2-4992-b070-c85423fa6d1e-kube-api-access-ppp9z\") pod \"maas-controller-c4874f67f-pckkq\" (UID: \"f3d819a3-29f2-4992-b070-c85423fa6d1e\") " pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:06:59.747773 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.747624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3d819a3-29f2-4992-b070-c85423fa6d1e-webhook-cert\") pod \"maas-controller-c4874f67f-pckkq\" (UID: \"f3d819a3-29f2-4992-b070-c85423fa6d1e\") " pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:06:59.750110 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.750086 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3d819a3-29f2-4992-b070-c85423fa6d1e-webhook-cert\") pod \"maas-controller-c4874f67f-pckkq\" (UID: \"f3d819a3-29f2-4992-b070-c85423fa6d1e\") " pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:06:59.757074 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.757047 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppp9z\" (UniqueName: \"kubernetes.io/projected/f3d819a3-29f2-4992-b070-c85423fa6d1e-kube-api-access-ppp9z\") pod \"maas-controller-c4874f67f-pckkq\" (UID: \"f3d819a3-29f2-4992-b070-c85423fa6d1e\") " pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:06:59.879228 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:06:59.879199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:07:00.012416 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:07:00.012391 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4874f67f-pckkq"] Jul 20 23:07:00.014200 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:07:00.014173 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d819a3_29f2_4992_b070_c85423fa6d1e.slice/crio-47a9040384b8f33128c3a01073c351acd70872a169105a61b4acffe464071209 WatchSource:0}: Error finding container 47a9040384b8f33128c3a01073c351acd70872a169105a61b4acffe464071209: Status 404 returned error can't find the container with id 47a9040384b8f33128c3a01073c351acd70872a169105a61b4acffe464071209 Jul 20 23:07:00.791742 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:07:00.791658 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-pckkq" event={"ID":"f3d819a3-29f2-4992-b070-c85423fa6d1e","Type":"ContainerStarted","Data":"733ab4980574ca9a541272322fe4ff050f2a59aac7386940f2d63b73607169df"} Jul 20 23:07:00.791742 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:07:00.791711 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4874f67f-pckkq" event={"ID":"f3d819a3-29f2-4992-b070-c85423fa6d1e","Type":"ContainerStarted","Data":"47a9040384b8f33128c3a01073c351acd70872a169105a61b4acffe464071209"} Jul 20 23:07:00.792149 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:07:00.791779 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:07:00.810616 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:07:00.810565 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c4874f67f-pckkq" podStartSLOduration=1.425124723 podStartE2EDuration="1.810549263s" totalImagesPullingTime="385.42454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:07:00.81027366 +0000 UTC m=+796.128625960" watchObservedRunningTime="2026-07-20 23:07:00.810549263 +0000 UTC m=+796.128901549" Jul 20 23:07:11.799267 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:07:11.799187 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4874f67f-pckkq" Jul 20 23:08:45.097040 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:08:45.096967 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:08:45.097040 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:08:45.097021 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:13:45.122795 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:13:45.122765 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:13:45.125379 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:13:45.123619 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:15:34.049061 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.049020 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk"] Jul 20 23:15:34.052584 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.052566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.055582 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.055559 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6f374799-kserve-self-signed-certs\"" Jul 20 23:15:34.063400 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.063374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk"] Jul 20 23:15:34.184336 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.184299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-model-cache\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.184527 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.184366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tmp-dir\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.184527 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.184442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-dshm\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.184527 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.184494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29pp\" (UniqueName: \"kubernetes.io/projected/712c01b0-a435-4d30-9e18-1f160c3dd4cd-kube-api-access-t29pp\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.184527 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.184522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-home\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.184728 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.184618 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tls-certs\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.286144 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.286103 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29pp\" (UniqueName: \"kubernetes.io/projected/712c01b0-a435-4d30-9e18-1f160c3dd4cd-kube-api-access-t29pp\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.286314 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.286170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-home\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.286314 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.286220 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tls-certs\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.286314 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.286281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-model-cache\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.286314 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.286304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tmp-dir\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.286574 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.286361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-dshm\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.286647 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.286628 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-home\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.286717 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.286695 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-model-cache\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.286778 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.286722 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tmp-dir\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.288563 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.288545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-dshm\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.288835 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.288817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tls-certs\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.294063 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.294039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29pp\" (UniqueName: \"kubernetes.io/projected/712c01b0-a435-4d30-9e18-1f160c3dd4cd-kube-api-access-t29pp\") pod \"e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.361839 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.361800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:34.499434 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:34.496936 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk"] Jul 20 23:15:34.500089 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:15:34.500056 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod712c01b0_a435_4d30_9e18_1f160c3dd4cd.slice/crio-6430190746c4c3eabe397d907dd1d604b0025a7b39919e859f83b01212b664e7 WatchSource:0}: Error finding container 6430190746c4c3eabe397d907dd1d604b0025a7b39919e859f83b01212b664e7: Status 404 returned error can't find the container with id 6430190746c4c3eabe397d907dd1d604b0025a7b39919e859f83b01212b664e7 Jul 20 23:15:35.107301 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:35.107270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" event={"ID":"712c01b0-a435-4d30-9e18-1f160c3dd4cd","Type":"ContainerStarted","Data":"a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e"} Jul 20 23:15:35.107682 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:35.107310 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" event={"ID":"712c01b0-a435-4d30-9e18-1f160c3dd4cd","Type":"ContainerStarted","Data":"6430190746c4c3eabe397d907dd1d604b0025a7b39919e859f83b01212b664e7"} Jul 20 23:15:35.124737 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:35.124682 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" podStartSLOduration=1.124666991 podStartE2EDuration="1.124666991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:15:35.122808908 +0000 UTC m=+1310.441161194" watchObservedRunningTime="2026-07-20 23:15:35.124666991 +0000 UTC m=+1310.443019276" Jul 20 23:15:43.507147 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.507112 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb"] Jul 20 23:15:43.511208 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.511189 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.513949 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.513921 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6f374799-kserve-self-signed-certs\"" Jul 20 23:15:43.520951 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.520923 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb"] Jul 20 23:15:43.667636 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.667598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bef2b17-c388-4fc5-b9d5-8536752b48de-tls-certs\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.667810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.667646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz68z\" (UniqueName: \"kubernetes.io/projected/3bef2b17-c388-4fc5-b9d5-8536752b48de-kube-api-access-jz68z\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.667810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.667742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-dshm\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.667810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.667771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-model-cache\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.667923 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.667810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-tmp-dir\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.667923 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.667839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-home\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.768970 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.768882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-dshm\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.768970 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.768925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-model-cache\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.769151 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.769047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-tmp-dir\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.769151 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.769091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-home\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.769151 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.769147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bef2b17-c388-4fc5-b9d5-8536752b48de-tls-certs\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.769264 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.769178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz68z\" (UniqueName: \"kubernetes.io/projected/3bef2b17-c388-4fc5-b9d5-8536752b48de-kube-api-access-jz68z\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.769418 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.769385 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-model-cache\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.769418 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.769408 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-tmp-dir\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.769647 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.769510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-home\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.771109 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.771089 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-dshm\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.771474 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.771440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bef2b17-c388-4fc5-b9d5-8536752b48de-tls-certs\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.778436 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.778411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz68z\" (UniqueName: \"kubernetes.io/projected/3bef2b17-c388-4fc5-b9d5-8536752b48de-kube-api-access-jz68z\") pod \"e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.823272 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.823225 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:43.955576 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:43.955545 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb"] Jul 20 23:15:43.957340 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:15:43.957308 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bef2b17_c388_4fc5_b9d5_8536752b48de.slice/crio-af6b50927ed22a3e59a57b857a41a0ea2f76cde19bcfbf38cc08f8682298006f WatchSource:0}: Error finding container af6b50927ed22a3e59a57b857a41a0ea2f76cde19bcfbf38cc08f8682298006f: Status 404 returned error can't find the container with id af6b50927ed22a3e59a57b857a41a0ea2f76cde19bcfbf38cc08f8682298006f Jul 20 23:15:44.149870 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:44.149831 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" event={"ID":"3bef2b17-c388-4fc5-b9d5-8536752b48de","Type":"ContainerStarted","Data":"87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a"} Jul 20 23:15:44.150083 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:44.149882 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" event={"ID":"3bef2b17-c388-4fc5-b9d5-8536752b48de","Type":"ContainerStarted","Data":"af6b50927ed22a3e59a57b857a41a0ea2f76cde19bcfbf38cc08f8682298006f"} Jul 20 23:15:44.168660 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:44.168609 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" podStartSLOduration=1.168591318 podStartE2EDuration="1.168591318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:15:44.166408666 +0000 UTC m=+1319.484760977" watchObservedRunningTime="2026-07-20 23:15:44.168591318 +0000 UTC m=+1319.486943675" Jul 20 23:15:44.362377 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:44.362331 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:44.362590 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:44.362392 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:44.375860 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:44.375830 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:45.166600 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:45.166570 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:15:53.823818 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:53.823772 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:53.824303 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:53.824057 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:53.836621 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:53.836593 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:15:54.208499 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:15:54.208470 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:16:13.548148 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:13.548108 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk"] Jul 20 23:16:14.659594 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:14.659550 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb"] Jul 20 23:16:28.570550 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.570507 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" podUID="712c01b0-a435-4d30-9e18-1f160c3dd4cd" containerName="main" containerID="cri-o://a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e" gracePeriod=45 Jul 20 23:16:28.822385 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.822316 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:16:28.988432 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988396 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-home\") pod \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " Jul 20 23:16:28.988603 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988448 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-model-cache\") pod \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " Jul 20 23:16:28.988603 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988488 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t29pp\" (UniqueName: \"kubernetes.io/projected/712c01b0-a435-4d30-9e18-1f160c3dd4cd-kube-api-access-t29pp\") pod \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " Jul 20 23:16:28.988603 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988549 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-dshm\") pod \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " Jul 20 23:16:28.988603 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tmp-dir\") pod \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " Jul 20 23:16:28.988771 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988629 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tls-certs\") pod \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\" (UID: \"712c01b0-a435-4d30-9e18-1f160c3dd4cd\") " Jul 20 23:16:28.988771 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988735 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "712c01b0-a435-4d30-9e18-1f160c3dd4cd" (UID: "712c01b0-a435-4d30-9e18-1f160c3dd4cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:16:28.988771 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988741 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-home" (OuterVolumeSpecName: "home") pod "712c01b0-a435-4d30-9e18-1f160c3dd4cd" (UID: "712c01b0-a435-4d30-9e18-1f160c3dd4cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:16:28.988926 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988882 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-home\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:28.988926 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988894 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-model-cache\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:28.988926 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.988886 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "712c01b0-a435-4d30-9e18-1f160c3dd4cd" (UID: "712c01b0-a435-4d30-9e18-1f160c3dd4cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:16:28.990735 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.990713 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-dshm" (OuterVolumeSpecName: "dshm") pod "712c01b0-a435-4d30-9e18-1f160c3dd4cd" (UID: "712c01b0-a435-4d30-9e18-1f160c3dd4cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:16:28.990856 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.990835 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712c01b0-a435-4d30-9e18-1f160c3dd4cd-kube-api-access-t29pp" (OuterVolumeSpecName: "kube-api-access-t29pp") pod "712c01b0-a435-4d30-9e18-1f160c3dd4cd" (UID: "712c01b0-a435-4d30-9e18-1f160c3dd4cd"). InnerVolumeSpecName "kube-api-access-t29pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:16:28.990933 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:28.990910 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "712c01b0-a435-4d30-9e18-1f160c3dd4cd" (UID: "712c01b0-a435-4d30-9e18-1f160c3dd4cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:16:29.089856 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.089772 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-dshm\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:29.089856 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.089803 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tmp-dir\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:29.089856 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.089817 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712c01b0-a435-4d30-9e18-1f160c3dd4cd-tls-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:29.089856 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.089829 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t29pp\" (UniqueName: \"kubernetes.io/projected/712c01b0-a435-4d30-9e18-1f160c3dd4cd-kube-api-access-t29pp\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:29.360246 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.360214 2574 generic.go:356] "Generic (PLEG): container finished" podID="712c01b0-a435-4d30-9e18-1f160c3dd4cd" containerID="a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e" exitCode=0 Jul 20 23:16:29.360409 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.360257 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" event={"ID":"712c01b0-a435-4d30-9e18-1f160c3dd4cd","Type":"ContainerDied","Data":"a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e"} Jul 20 23:16:29.360409 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.360280 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" Jul 20 23:16:29.360409 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.360299 2574 scope.go:117] "RemoveContainer" containerID="a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e" Jul 20 23:16:29.360409 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.360288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk" event={"ID":"712c01b0-a435-4d30-9e18-1f160c3dd4cd","Type":"ContainerDied","Data":"6430190746c4c3eabe397d907dd1d604b0025a7b39919e859f83b01212b664e7"} Jul 20 23:16:29.369198 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.369175 2574 scope.go:117] "RemoveContainer" containerID="a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e" Jul 20 23:16:29.369485 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:16:29.369446 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e\": container with ID starting with a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e not found: ID does not exist" containerID="a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e" Jul 20 23:16:29.369569 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.369494 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e"} err="failed to get container status \"a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e\": rpc error: code = NotFound desc = could not find container \"a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e\": container with ID starting with a35eda68d3544f958938499773b3ca798dfed042aa1e1f7212c038a87f1c1a4e not found: ID does not exist" Jul 20 23:16:29.378575 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.378552 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk"] Jul 20 23:16:29.381943 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.381922 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6f374799-kserve-7dc4cffcd7-2v2vk"] Jul 20 23:16:29.681695 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.681595 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" podUID="3bef2b17-c388-4fc5-b9d5-8536752b48de" containerName="main" containerID="cri-o://87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a" gracePeriod=45 Jul 20 23:16:29.946112 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:29.946053 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:16:30.098339 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.098309 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bef2b17-c388-4fc5-b9d5-8536752b48de-tls-certs\") pod \"3bef2b17-c388-4fc5-b9d5-8536752b48de\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " Jul 20 23:16:30.098526 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.098396 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-model-cache\") pod \"3bef2b17-c388-4fc5-b9d5-8536752b48de\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " Jul 20 23:16:30.098526 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.098425 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-dshm\") pod \"3bef2b17-c388-4fc5-b9d5-8536752b48de\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " Jul 20 23:16:30.098526 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.098470 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-home\") pod \"3bef2b17-c388-4fc5-b9d5-8536752b48de\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " Jul 20 23:16:30.098526 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.098504 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz68z\" (UniqueName: \"kubernetes.io/projected/3bef2b17-c388-4fc5-b9d5-8536752b48de-kube-api-access-jz68z\") pod \"3bef2b17-c388-4fc5-b9d5-8536752b48de\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " Jul 20 23:16:30.098526 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.098522 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-tmp-dir\") pod \"3bef2b17-c388-4fc5-b9d5-8536752b48de\" (UID: \"3bef2b17-c388-4fc5-b9d5-8536752b48de\") " Jul 20 23:16:30.098833 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.098672 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-model-cache" (OuterVolumeSpecName: "model-cache") pod "3bef2b17-c388-4fc5-b9d5-8536752b48de" (UID: "3bef2b17-c388-4fc5-b9d5-8536752b48de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:16:30.098833 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.098740 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-model-cache\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:30.098833 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.098773 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-home" (OuterVolumeSpecName: "home") pod "3bef2b17-c388-4fc5-b9d5-8536752b48de" (UID: "3bef2b17-c388-4fc5-b9d5-8536752b48de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:16:30.099099 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.099072 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3bef2b17-c388-4fc5-b9d5-8536752b48de" (UID: "3bef2b17-c388-4fc5-b9d5-8536752b48de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:16:30.100601 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.100575 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-dshm" (OuterVolumeSpecName: "dshm") pod "3bef2b17-c388-4fc5-b9d5-8536752b48de" (UID: "3bef2b17-c388-4fc5-b9d5-8536752b48de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:16:30.100703 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.100601 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bef2b17-c388-4fc5-b9d5-8536752b48de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3bef2b17-c388-4fc5-b9d5-8536752b48de" (UID: "3bef2b17-c388-4fc5-b9d5-8536752b48de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:16:30.100703 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.100649 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bef2b17-c388-4fc5-b9d5-8536752b48de-kube-api-access-jz68z" (OuterVolumeSpecName: "kube-api-access-jz68z") pod "3bef2b17-c388-4fc5-b9d5-8536752b48de" (UID: "3bef2b17-c388-4fc5-b9d5-8536752b48de"). InnerVolumeSpecName "kube-api-access-jz68z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:16:30.199794 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.199690 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-dshm\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:30.199794 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.199738 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-home\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:30.199794 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.199747 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz68z\" (UniqueName: \"kubernetes.io/projected/3bef2b17-c388-4fc5-b9d5-8536752b48de-kube-api-access-jz68z\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:30.199794 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.199757 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bef2b17-c388-4fc5-b9d5-8536752b48de-tmp-dir\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:30.199794 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.199766 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bef2b17-c388-4fc5-b9d5-8536752b48de-tls-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:16:30.366349 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.366318 2574 generic.go:356] "Generic (PLEG): container finished" podID="3bef2b17-c388-4fc5-b9d5-8536752b48de" containerID="87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a" exitCode=0 Jul 20 23:16:30.366549 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.366360 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" Jul 20 23:16:30.366549 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.366403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" event={"ID":"3bef2b17-c388-4fc5-b9d5-8536752b48de","Type":"ContainerDied","Data":"87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a"} Jul 20 23:16:30.366549 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.366440 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb" event={"ID":"3bef2b17-c388-4fc5-b9d5-8536752b48de","Type":"ContainerDied","Data":"af6b50927ed22a3e59a57b857a41a0ea2f76cde19bcfbf38cc08f8682298006f"} Jul 20 23:16:30.366549 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.366476 2574 scope.go:117] "RemoveContainer" containerID="87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a" Jul 20 23:16:30.377345 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.377318 2574 scope.go:117] "RemoveContainer" containerID="87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a" Jul 20 23:16:30.377668 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:16:30.377641 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a\": container with ID starting with 87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a not found: ID does not exist" containerID="87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a" Jul 20 23:16:30.377754 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.377681 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a"} err="failed to get container status \"87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a\": rpc error: code = NotFound desc = could not find container \"87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a\": container with ID starting with 87ece1bb78f3e443368a2711538719f1884b2bf54f2f44ad61b7c315f534227a not found: ID does not exist" Jul 20 23:16:30.388688 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.388663 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb"] Jul 20 23:16:30.392566 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:30.392545 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6f374799-kserve-6d6d5cdd47-m55fb"] Jul 20 23:16:31.227978 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:31.227932 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bef2b17-c388-4fc5-b9d5-8536752b48de" path="/var/lib/kubelet/pods/3bef2b17-c388-4fc5-b9d5-8536752b48de/volumes" Jul 20 23:16:31.228526 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:16:31.228507 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="712c01b0-a435-4d30-9e18-1f160c3dd4cd" path="/var/lib/kubelet/pods/712c01b0-a435-4d30-9e18-1f160c3dd4cd/volumes" Jul 20 23:18:45.149681 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:18:45.149560 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:18:45.153705 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:18:45.150794 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:19:57.116468 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.116424 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw"] Jul 20 23:19:57.116950 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.116814 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bef2b17-c388-4fc5-b9d5-8536752b48de" containerName="main" Jul 20 23:19:57.116950 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.116827 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bef2b17-c388-4fc5-b9d5-8536752b48de" containerName="main" Jul 20 23:19:57.116950 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.116846 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="712c01b0-a435-4d30-9e18-1f160c3dd4cd" containerName="main" Jul 20 23:19:57.116950 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.116852 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="712c01b0-a435-4d30-9e18-1f160c3dd4cd" containerName="main" Jul 20 23:19:57.116950 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.116908 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bef2b17-c388-4fc5-b9d5-8536752b48de" containerName="main" Jul 20 23:19:57.116950 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.116920 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="712c01b0-a435-4d30-9e18-1f160c3dd4cd" containerName="main" Jul 20 23:19:57.120095 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.120077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.123129 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.123110 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e58bad50-openshift-default-dockercfg-4nxxm\"" Jul 20 23:19:57.145312 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.145283 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw"] Jul 20 23:19:57.217665 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.217630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-credential-socket\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.217861 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.217674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-podinfo\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.217861 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.217697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-data\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.217861 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.217807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-envoy\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.218035 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.217866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-socket\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.218035 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.217916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-certs\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.218035 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.217955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9r26\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-kube-api-access-m9r26\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.218035 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.217990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99cf074-b01f-4958-8721-bb2e35f692a5-istiod-ca-cert\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.218035 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.218015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-token\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.319719 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.319687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99cf074-b01f-4958-8721-bb2e35f692a5-istiod-ca-cert\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.319719 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.319723 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-token\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.319929 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.319767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-credential-socket\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.319929 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.319791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-podinfo\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320095 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.319933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-data\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320095 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.319994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-envoy\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320095 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.320052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-socket\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320095 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.320090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-certs\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320291 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.320130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9r26\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-kube-api-access-m9r26\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320435 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.320405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-credential-socket\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320680 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.320657 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99cf074-b01f-4958-8721-bb2e35f692a5-istiod-ca-cert\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320779 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.320709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-socket\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320779 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.320744 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-data\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.320960 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.320941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-certs\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.322701 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.322667 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-podinfo\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.323150 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.323132 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-envoy\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.329178 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.329148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-token\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.329420 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.329397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9r26\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-kube-api-access-m9r26\") pod \"e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.430781 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.430679 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:57.571628 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.571599 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw"] Jul 20 23:19:57.574451 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:19:57.574424 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99cf074_b01f_4958_8721_bb2e35f692a5.slice/crio-d0b83e3d6f85541aaa1bd050a5ee85bac4d070e281180b571621958fca2300fa WatchSource:0}: Error finding container d0b83e3d6f85541aaa1bd050a5ee85bac4d070e281180b571621958fca2300fa: Status 404 returned error can't find the container with id d0b83e3d6f85541aaa1bd050a5ee85bac4d070e281180b571621958fca2300fa Jul 20 23:19:57.577066 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.577028 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:19:57.577151 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.577096 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:19:57.577151 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:57.577124 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:19:58.309904 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:58.309865 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" event={"ID":"e99cf074-b01f-4958-8721-bb2e35f692a5","Type":"ContainerStarted","Data":"e1414ce773423cfb9afae275d71608eff162ff9a3f7c2b96263640fd5756a2b0"} Jul 20 23:19:58.310284 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:58.309924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" event={"ID":"e99cf074-b01f-4958-8721-bb2e35f692a5","Type":"ContainerStarted","Data":"d0b83e3d6f85541aaa1bd050a5ee85bac4d070e281180b571621958fca2300fa"} Jul 20 23:19:58.331533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:58.331485 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" podStartSLOduration=1.3314692209999999 podStartE2EDuration="1.331469221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:19:58.329287411 +0000 UTC m=+1573.647639698" watchObservedRunningTime="2026-07-20 23:19:58.331469221 +0000 UTC m=+1573.649821499" Jul 20 23:19:58.430872 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:58.430837 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:58.435681 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:58.435652 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:59.314506 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.314478 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:59.315496 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.315451 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:19:59.730697 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.730661 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh"] Jul 20 23:19:59.734419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.734393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:19:59.736978 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.736952 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e58bad50-serving-cert\"" Jul 20 23:19:59.750484 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.750434 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh"] Jul 20 23:19:59.843105 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.843067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946eeeca-7dab-40ac-a276-6433fb79b91a-openshift-service-ca\") pod \"maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:19:59.843299 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.843139 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946eeeca-7dab-40ac-a276-6433fb79b91a-maas-api-tls\") pod \"maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:19:59.843299 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.843173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7gfh\" (UniqueName: \"kubernetes.io/projected/946eeeca-7dab-40ac-a276-6433fb79b91a-kube-api-access-v7gfh\") pod \"maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:19:59.944400 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.944357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946eeeca-7dab-40ac-a276-6433fb79b91a-maas-api-tls\") pod \"maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:19:59.944400 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.944402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7gfh\" (UniqueName: \"kubernetes.io/projected/946eeeca-7dab-40ac-a276-6433fb79b91a-kube-api-access-v7gfh\") pod \"maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:19:59.944626 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.944482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946eeeca-7dab-40ac-a276-6433fb79b91a-openshift-service-ca\") pod \"maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:19:59.945175 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.945152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946eeeca-7dab-40ac-a276-6433fb79b91a-openshift-service-ca\") pod \"maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:19:59.946781 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.946761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946eeeca-7dab-40ac-a276-6433fb79b91a-maas-api-tls\") pod \"maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:19:59.956999 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:19:59.956972 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7gfh\" (UniqueName: \"kubernetes.io/projected/946eeeca-7dab-40ac-a276-6433fb79b91a-kube-api-access-v7gfh\") pod \"maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:20:00.045354 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:00.045260 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:20:00.191106 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:00.191082 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh"] Jul 20 23:20:00.193042 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:20:00.193012 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946eeeca_7dab_40ac_a276_6433fb79b91a.slice/crio-d4ec3f3b47f60e781ce899087913ad8275c673b214d421bf28e87efff5924b61 WatchSource:0}: Error finding container d4ec3f3b47f60e781ce899087913ad8275c673b214d421bf28e87efff5924b61: Status 404 returned error can't find the container with id d4ec3f3b47f60e781ce899087913ad8275c673b214d421bf28e87efff5924b61 Jul 20 23:20:00.194406 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:00.194387 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 23:20:00.320067 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:00.319976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" event={"ID":"946eeeca-7dab-40ac-a276-6433fb79b91a","Type":"ContainerStarted","Data":"d4ec3f3b47f60e781ce899087913ad8275c673b214d421bf28e87efff5924b61"} Jul 20 23:20:01.325624 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:01.325584 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" event={"ID":"946eeeca-7dab-40ac-a276-6433fb79b91a","Type":"ContainerStarted","Data":"c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d"} Jul 20 23:20:01.345880 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:01.345819 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" podStartSLOduration=1.872580074 podStartE2EDuration="2.345803715s" totalImagesPullingTime="473.223641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:20:01.343855755 +0000 UTC m=+1576.662208039" watchObservedRunningTime="2026-07-20 23:20:01.345803715 +0000 UTC m=+1576.664156001" Jul 20 23:20:02.330792 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:02.330746 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:20:08.341743 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:08.341711 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:20:13.385225 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:13.385193 2574 generic.go:356] "Generic (PLEG): container finished" podID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerID="e1414ce773423cfb9afae275d71608eff162ff9a3f7c2b96263640fd5756a2b0" exitCode=0 Jul 20 23:20:13.385761 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:13.385269 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" event={"ID":"e99cf074-b01f-4958-8721-bb2e35f692a5","Type":"ContainerDied","Data":"e1414ce773423cfb9afae275d71608eff162ff9a3f7c2b96263640fd5756a2b0"} Jul 20 23:20:13.385826 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:13.385789 2574 scope.go:117] "RemoveContainer" containerID="e1414ce773423cfb9afae275d71608eff162ff9a3f7c2b96263640fd5756a2b0" Jul 20 23:20:13.386816 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:13.386784 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:20:13.386915 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:13.386869 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:20:13.386975 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:13.386921 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:20:14.314395 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:14.314356 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:20:14.392353 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:14.392318 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" event={"ID":"e99cf074-b01f-4958-8721-bb2e35f692a5","Type":"ContainerStarted","Data":"c393940f8c7d440c90dc998c7c7c6d95df0637832686209d78a674a7030b4dc8"} Jul 20 23:20:14.431119 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:14.431084 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:20:14.432200 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:14.432170 2574 patch_prober.go:28] interesting pod/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 20 23:20:14.432341 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:14.432239 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 20 23:20:15.431108 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:15.431074 2574 patch_prober.go:28] interesting pod/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 20 23:20:15.431512 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:15.431126 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 20 23:20:16.431120 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:16.431085 2574 patch_prober.go:28] interesting pod/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 20 23:20:16.431540 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:16.431138 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 20 23:20:17.431427 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:17.431393 2574 patch_prober.go:28] interesting pod/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 20 23:20:17.431821 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:17.431447 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 20 23:20:18.435249 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:18.435218 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:20:18.435708 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:18.435500 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:20:18.436317 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:18.436299 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:20:35.561841 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:35.561810 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh"] Jul 20 23:20:35.562269 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:35.562016 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" podUID="946eeeca-7dab-40ac-a276-6433fb79b91a" containerName="maas-api" containerID="cri-o://c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d" gracePeriod=30 Jul 20 23:20:35.819742 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:35.819686 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:20:35.980082 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:35.980046 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946eeeca-7dab-40ac-a276-6433fb79b91a-openshift-service-ca\") pod \"946eeeca-7dab-40ac-a276-6433fb79b91a\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " Jul 20 23:20:35.980280 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:35.980187 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7gfh\" (UniqueName: \"kubernetes.io/projected/946eeeca-7dab-40ac-a276-6433fb79b91a-kube-api-access-v7gfh\") pod \"946eeeca-7dab-40ac-a276-6433fb79b91a\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " Jul 20 23:20:35.980280 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:35.980216 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946eeeca-7dab-40ac-a276-6433fb79b91a-maas-api-tls\") pod \"946eeeca-7dab-40ac-a276-6433fb79b91a\" (UID: \"946eeeca-7dab-40ac-a276-6433fb79b91a\") " Jul 20 23:20:35.980514 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:35.980454 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946eeeca-7dab-40ac-a276-6433fb79b91a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "946eeeca-7dab-40ac-a276-6433fb79b91a" (UID: "946eeeca-7dab-40ac-a276-6433fb79b91a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:20:35.982255 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:35.982225 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946eeeca-7dab-40ac-a276-6433fb79b91a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "946eeeca-7dab-40ac-a276-6433fb79b91a" (UID: "946eeeca-7dab-40ac-a276-6433fb79b91a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:20:35.982370 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:35.982327 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946eeeca-7dab-40ac-a276-6433fb79b91a-kube-api-access-v7gfh" (OuterVolumeSpecName: "kube-api-access-v7gfh") pod "946eeeca-7dab-40ac-a276-6433fb79b91a" (UID: "946eeeca-7dab-40ac-a276-6433fb79b91a"). InnerVolumeSpecName "kube-api-access-v7gfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:20:36.081537 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.081409 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7gfh\" (UniqueName: \"kubernetes.io/projected/946eeeca-7dab-40ac-a276-6433fb79b91a-kube-api-access-v7gfh\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:36.081537 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.081445 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/946eeeca-7dab-40ac-a276-6433fb79b91a-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:36.081537 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.081492 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/946eeeca-7dab-40ac-a276-6433fb79b91a-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:36.492615 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.492576 2574 generic.go:356] "Generic (PLEG): container finished" podID="946eeeca-7dab-40ac-a276-6433fb79b91a" containerID="c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d" exitCode=0 Jul 20 23:20:36.492799 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.492629 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" Jul 20 23:20:36.492799 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.492624 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" event={"ID":"946eeeca-7dab-40ac-a276-6433fb79b91a","Type":"ContainerDied","Data":"c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d"} Jul 20 23:20:36.492799 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.492771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh" event={"ID":"946eeeca-7dab-40ac-a276-6433fb79b91a","Type":"ContainerDied","Data":"d4ec3f3b47f60e781ce899087913ad8275c673b214d421bf28e87efff5924b61"} Jul 20 23:20:36.492973 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.492809 2574 scope.go:117] "RemoveContainer" containerID="c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d" Jul 20 23:20:36.501623 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.501603 2574 scope.go:117] "RemoveContainer" containerID="c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d" Jul 20 23:20:36.501896 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:20:36.501875 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d\": container with ID starting with c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d not found: ID does not exist" containerID="c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d" Jul 20 23:20:36.501960 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.501904 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d"} err="failed to get container status \"c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d\": rpc error: code = NotFound desc = could not find container \"c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d\": container with ID starting with c1d0e50171ba88dd4789ec769b56dfa1e048933a57c1666b21f4d14e8cd0713d not found: ID does not exist" Jul 20 23:20:36.513844 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.513816 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh"] Jul 20 23:20:36.515734 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:36.515709 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58bad50-5dbfc7c599-vjshh"] Jul 20 23:20:37.227687 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:37.227615 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="946eeeca-7dab-40ac-a276-6433fb79b91a" path="/var/lib/kubelet/pods/946eeeca-7dab-40ac-a276-6433fb79b91a/volumes" Jul 20 23:20:40.379851 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:40.379809 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw"] Jul 20 23:20:40.380320 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:40.380120 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" containerID="cri-o://c393940f8c7d440c90dc998c7c7c6d95df0637832686209d78a674a7030b4dc8" gracePeriod=30 Jul 20 23:20:43.781081 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.781042 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74"] Jul 20 23:20:43.781495 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.781400 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946eeeca-7dab-40ac-a276-6433fb79b91a" containerName="maas-api" Jul 20 23:20:43.781495 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.781410 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="946eeeca-7dab-40ac-a276-6433fb79b91a" containerName="maas-api" Jul 20 23:20:43.781641 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.781515 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="946eeeca-7dab-40ac-a276-6433fb79b91a" containerName="maas-api" Jul 20 23:20:43.785763 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.785738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:43.788431 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.788402 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5ae564de-serving-cert\"" Jul 20 23:20:43.793516 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.793490 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74"] Jul 20 23:20:43.954351 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.954313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1afbbb9e-13ea-4ead-bb76-b579f397fcee-maas-api-tls\") pod \"maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:43.954561 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.954378 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfhsp\" (UniqueName: \"kubernetes.io/projected/1afbbb9e-13ea-4ead-bb76-b579f397fcee-kube-api-access-nfhsp\") pod \"maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:43.954561 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:43.954404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1afbbb9e-13ea-4ead-bb76-b579f397fcee-openshift-service-ca\") pod \"maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:44.055264 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:44.055162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1afbbb9e-13ea-4ead-bb76-b579f397fcee-maas-api-tls\") pod \"maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:44.055264 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:44.055246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfhsp\" (UniqueName: \"kubernetes.io/projected/1afbbb9e-13ea-4ead-bb76-b579f397fcee-kube-api-access-nfhsp\") pod \"maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:44.055550 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:44.055284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1afbbb9e-13ea-4ead-bb76-b579f397fcee-openshift-service-ca\") pod \"maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:44.056037 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:44.056012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1afbbb9e-13ea-4ead-bb76-b579f397fcee-openshift-service-ca\") pod \"maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:44.057635 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:44.057618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1afbbb9e-13ea-4ead-bb76-b579f397fcee-maas-api-tls\") pod \"maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:44.066763 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:44.066738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfhsp\" (UniqueName: \"kubernetes.io/projected/1afbbb9e-13ea-4ead-bb76-b579f397fcee-kube-api-access-nfhsp\") pod \"maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:44.098781 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:44.098749 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:44.239829 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:44.239802 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74"] Jul 20 23:20:44.241929 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:20:44.241903 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afbbb9e_13ea_4ead_bb76_b579f397fcee.slice/crio-a86f744f8acd34c0f9f770d19d4a7e7c5cda86ae8ae69791974e745187655697 WatchSource:0}: Error finding container a86f744f8acd34c0f9f770d19d4a7e7c5cda86ae8ae69791974e745187655697: Status 404 returned error can't find the container with id a86f744f8acd34c0f9f770d19d4a7e7c5cda86ae8ae69791974e745187655697 Jul 20 23:20:44.538055 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:44.538018 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" event={"ID":"1afbbb9e-13ea-4ead-bb76-b579f397fcee","Type":"ContainerStarted","Data":"a86f744f8acd34c0f9f770d19d4a7e7c5cda86ae8ae69791974e745187655697"} Jul 20 23:20:45.545082 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.545042 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" event={"ID":"1afbbb9e-13ea-4ead-bb76-b579f397fcee","Type":"ContainerStarted","Data":"5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8"} Jul 20 23:20:45.545515 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.545181 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:20:45.547411 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.547383 2574 generic.go:356] "Generic (PLEG): container finished" podID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerID="c393940f8c7d440c90dc998c7c7c6d95df0637832686209d78a674a7030b4dc8" exitCode=0 Jul 20 23:20:45.547544 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.547422 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" event={"ID":"e99cf074-b01f-4958-8721-bb2e35f692a5","Type":"ContainerDied","Data":"c393940f8c7d440c90dc998c7c7c6d95df0637832686209d78a674a7030b4dc8"} Jul 20 23:20:45.547544 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.547485 2574 scope.go:117] "RemoveContainer" containerID="e1414ce773423cfb9afae275d71608eff162ff9a3f7c2b96263640fd5756a2b0" Jul 20 23:20:45.564758 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.564279 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" podStartSLOduration=2.224993847 podStartE2EDuration="2.564259813s" totalImagesPullingTime="339.265966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:20:45.563721536 +0000 UTC m=+1620.882073823" watchObservedRunningTime="2026-07-20 23:20:45.564259813 +0000 UTC m=+1620.882612100" Jul 20 23:20:45.641396 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.641337 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:20:45.772032 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.771998 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-data\") pod \"e99cf074-b01f-4958-8721-bb2e35f692a5\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " Jul 20 23:20:45.772204 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772052 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-credential-socket\") pod \"e99cf074-b01f-4958-8721-bb2e35f692a5\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " Jul 20 23:20:45.772204 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772075 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-socket\") pod \"e99cf074-b01f-4958-8721-bb2e35f692a5\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " Jul 20 23:20:45.772204 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772093 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-certs\") pod \"e99cf074-b01f-4958-8721-bb2e35f692a5\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " Jul 20 23:20:45.772204 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772123 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9r26\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-kube-api-access-m9r26\") pod \"e99cf074-b01f-4958-8721-bb2e35f692a5\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " Jul 20 23:20:45.772204 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772148 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-envoy\") pod \"e99cf074-b01f-4958-8721-bb2e35f692a5\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " Jul 20 23:20:45.772204 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772166 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-podinfo\") pod \"e99cf074-b01f-4958-8721-bb2e35f692a5\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " Jul 20 23:20:45.772204 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772182 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-token\") pod \"e99cf074-b01f-4958-8721-bb2e35f692a5\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " Jul 20 23:20:45.772597 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772214 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99cf074-b01f-4958-8721-bb2e35f692a5-istiod-ca-cert\") pod \"e99cf074-b01f-4958-8721-bb2e35f692a5\" (UID: \"e99cf074-b01f-4958-8721-bb2e35f692a5\") " Jul 20 23:20:45.772597 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772338 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e99cf074-b01f-4958-8721-bb2e35f692a5" (UID: "e99cf074-b01f-4958-8721-bb2e35f692a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:20:45.772597 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772413 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e99cf074-b01f-4958-8721-bb2e35f692a5" (UID: "e99cf074-b01f-4958-8721-bb2e35f692a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:20:45.772597 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772529 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-credential-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:45.772597 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772548 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:45.772799 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772755 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e99cf074-b01f-4958-8721-bb2e35f692a5" (UID: "e99cf074-b01f-4958-8721-bb2e35f692a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:20:45.772799 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772783 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99cf074-b01f-4958-8721-bb2e35f692a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e99cf074-b01f-4958-8721-bb2e35f692a5" (UID: "e99cf074-b01f-4958-8721-bb2e35f692a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:20:45.772967 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.772945 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "e99cf074-b01f-4958-8721-bb2e35f692a5" (UID: "e99cf074-b01f-4958-8721-bb2e35f692a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:20:45.774565 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.774537 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e99cf074-b01f-4958-8721-bb2e35f692a5" (UID: "e99cf074-b01f-4958-8721-bb2e35f692a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:20:45.774565 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.774544 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "e99cf074-b01f-4958-8721-bb2e35f692a5" (UID: "e99cf074-b01f-4958-8721-bb2e35f692a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:20:45.774921 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.774903 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-kube-api-access-m9r26" (OuterVolumeSpecName: "kube-api-access-m9r26") pod "e99cf074-b01f-4958-8721-bb2e35f692a5" (UID: "e99cf074-b01f-4958-8721-bb2e35f692a5"). InnerVolumeSpecName "kube-api-access-m9r26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:20:45.774994 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.774909 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e99cf074-b01f-4958-8721-bb2e35f692a5" (UID: "e99cf074-b01f-4958-8721-bb2e35f692a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 23:20:45.873147 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.873107 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-data\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:45.873147 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.873139 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-workload-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:45.873147 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.873149 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9r26\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-kube-api-access-m9r26\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:45.873378 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.873159 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-envoy\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:45.873378 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.873168 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-podinfo\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:45.873378 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.873176 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99cf074-b01f-4958-8721-bb2e35f692a5-istio-token\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:45.873378 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:45.873184 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99cf074-b01f-4958-8721-bb2e35f692a5-istiod-ca-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:20:46.553413 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:46.553331 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" Jul 20 23:20:46.553413 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:46.553376 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw" event={"ID":"e99cf074-b01f-4958-8721-bb2e35f692a5","Type":"ContainerDied","Data":"d0b83e3d6f85541aaa1bd050a5ee85bac4d070e281180b571621958fca2300fa"} Jul 20 23:20:46.553924 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:46.553433 2574 scope.go:117] "RemoveContainer" containerID="c393940f8c7d440c90dc998c7c7c6d95df0637832686209d78a674a7030b4dc8" Jul 20 23:20:46.574288 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:46.574258 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw"] Jul 20 23:20:46.579006 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:46.578977 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e58bad50-openshift-default-798cd75d6c-j6dmw"] Jul 20 23:20:47.228382 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:47.228347 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" path="/var/lib/kubelet/pods/e99cf074-b01f-4958-8721-bb2e35f692a5/volumes" Jul 20 23:20:51.559135 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:20:51.559103 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:21:19.153493 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.153439 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74"] Jul 20 23:21:19.154060 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.153709 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" podUID="1afbbb9e-13ea-4ead-bb76-b579f397fcee" containerName="maas-api" containerID="cri-o://5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8" gracePeriod=30 Jul 20 23:21:19.410557 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.410494 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:21:19.455583 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.455548 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1afbbb9e-13ea-4ead-bb76-b579f397fcee-openshift-service-ca\") pod \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " Jul 20 23:21:19.455772 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.455598 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1afbbb9e-13ea-4ead-bb76-b579f397fcee-maas-api-tls\") pod \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " Jul 20 23:21:19.455772 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.455656 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfhsp\" (UniqueName: \"kubernetes.io/projected/1afbbb9e-13ea-4ead-bb76-b579f397fcee-kube-api-access-nfhsp\") pod \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\" (UID: \"1afbbb9e-13ea-4ead-bb76-b579f397fcee\") " Jul 20 23:21:19.456023 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.455996 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afbbb9e-13ea-4ead-bb76-b579f397fcee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1afbbb9e-13ea-4ead-bb76-b579f397fcee" (UID: "1afbbb9e-13ea-4ead-bb76-b579f397fcee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:21:19.457776 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.457750 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1afbbb9e-13ea-4ead-bb76-b579f397fcee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1afbbb9e-13ea-4ead-bb76-b579f397fcee" (UID: "1afbbb9e-13ea-4ead-bb76-b579f397fcee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:21:19.458768 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.458743 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afbbb9e-13ea-4ead-bb76-b579f397fcee-kube-api-access-nfhsp" (OuterVolumeSpecName: "kube-api-access-nfhsp") pod "1afbbb9e-13ea-4ead-bb76-b579f397fcee" (UID: "1afbbb9e-13ea-4ead-bb76-b579f397fcee"). InnerVolumeSpecName "kube-api-access-nfhsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:21:19.557046 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.557009 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1afbbb9e-13ea-4ead-bb76-b579f397fcee-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:21:19.557046 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.557041 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1afbbb9e-13ea-4ead-bb76-b579f397fcee-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:21:19.557046 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.557051 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfhsp\" (UniqueName: \"kubernetes.io/projected/1afbbb9e-13ea-4ead-bb76-b579f397fcee-kube-api-access-nfhsp\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:21:19.710913 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.710823 2574 generic.go:356] "Generic (PLEG): container finished" podID="1afbbb9e-13ea-4ead-bb76-b579f397fcee" containerID="5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8" exitCode=0 Jul 20 23:21:19.710913 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.710864 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" event={"ID":"1afbbb9e-13ea-4ead-bb76-b579f397fcee","Type":"ContainerDied","Data":"5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8"} Jul 20 23:21:19.710913 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.710912 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" event={"ID":"1afbbb9e-13ea-4ead-bb76-b579f397fcee","Type":"ContainerDied","Data":"a86f744f8acd34c0f9f770d19d4a7e7c5cda86ae8ae69791974e745187655697"} Jul 20 23:21:19.711177 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.710867 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74" Jul 20 23:21:19.711177 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.710941 2574 scope.go:117] "RemoveContainer" containerID="5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8" Jul 20 23:21:19.720291 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.720272 2574 scope.go:117] "RemoveContainer" containerID="5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8" Jul 20 23:21:19.720640 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:21:19.720618 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8\": container with ID starting with 5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8 not found: ID does not exist" containerID="5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8" Jul 20 23:21:19.720695 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.720650 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8"} err="failed to get container status \"5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8\": rpc error: code = NotFound desc = could not find container \"5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8\": container with ID starting with 5d9fc9f07a6f0b153af34e3f4c0d26c1baa6a6275c1bb9237e653a7a837c49d8 not found: ID does not exist" Jul 20 23:21:19.733312 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.733286 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74"] Jul 20 23:21:19.738304 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:19.738279 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5ae564de-6fcd8556c-9bw74"] Jul 20 23:21:21.227814 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:21.227779 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1afbbb9e-13ea-4ead-bb76-b579f397fcee" path="/var/lib/kubelet/pods/1afbbb9e-13ea-4ead-bb76-b579f397fcee/volumes" Jul 20 23:21:24.650796 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.650758 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp"] Jul 20 23:21:24.651283 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.651254 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1afbbb9e-13ea-4ead-bb76-b579f397fcee" containerName="maas-api" Jul 20 23:21:24.651283 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.651273 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1afbbb9e-13ea-4ead-bb76-b579f397fcee" containerName="maas-api" Jul 20 23:21:24.651381 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.651290 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" Jul 20 23:21:24.651381 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.651298 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" Jul 20 23:21:24.651449 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.651394 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1afbbb9e-13ea-4ead-bb76-b579f397fcee" containerName="maas-api" Jul 20 23:21:24.651449 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.651406 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" Jul 20 23:21:24.651449 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.651412 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" Jul 20 23:21:24.656027 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.656009 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.658801 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.658774 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-421a7b46-openshift-default-dockercfg-gwdmg\"" Jul 20 23:21:24.665574 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.665545 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp"] Jul 20 23:21:24.805133 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.805090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-envoy\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.805133 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.805132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-data\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.805426 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.805201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116f786f-6f8f-4f82-b476-86993912e899-istio-podinfo\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.805426 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.805240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116f786f-6f8f-4f82-b476-86993912e899-istiod-ca-cert\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.805426 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.805274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-socket\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.805426 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.805297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-credential-socket\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.805426 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.805312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n22gh\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-kube-api-access-n22gh\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.805426 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.805370 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-istio-token\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.805426 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.805406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-certs\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906139 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116f786f-6f8f-4f82-b476-86993912e899-istio-podinfo\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906139 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116f786f-6f8f-4f82-b476-86993912e899-istiod-ca-cert\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906368 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-socket\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906368 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-credential-socket\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906368 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n22gh\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-kube-api-access-n22gh\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906368 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-istio-token\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906368 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906330 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-certs\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906701 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-envoy\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906701 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-data\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906701 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-socket\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906834 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906727 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-credential-socket\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906881 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-data\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.906881 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.906852 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-certs\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.907077 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.907059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116f786f-6f8f-4f82-b476-86993912e899-istiod-ca-cert\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.908616 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.908594 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-envoy\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.908723 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.908660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116f786f-6f8f-4f82-b476-86993912e899-istio-podinfo\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.915069 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.915042 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-istio-token\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.915412 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.915395 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n22gh\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-kube-api-access-n22gh\") pod \"e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:24.968443 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:24.968406 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:25.111366 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:25.111330 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp"] Jul 20 23:21:25.113178 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:21:25.113150 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116f786f_6f8f_4f82_b476_86993912e899.slice/crio-a1f4f7abfedf16564f2ed0f11a88468e991a75211e5f01a19a21353e138f69a1 WatchSource:0}: Error finding container a1f4f7abfedf16564f2ed0f11a88468e991a75211e5f01a19a21353e138f69a1: Status 404 returned error can't find the container with id a1f4f7abfedf16564f2ed0f11a88468e991a75211e5f01a19a21353e138f69a1 Jul 20 23:21:25.115199 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:25.115164 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:21:25.115274 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:25.115231 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:21:25.115274 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:25.115258 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:21:25.740722 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:25.740685 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" event={"ID":"116f786f-6f8f-4f82-b476-86993912e899","Type":"ContainerStarted","Data":"0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809"} Jul 20 23:21:25.741146 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:25.740744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" event={"ID":"116f786f-6f8f-4f82-b476-86993912e899","Type":"ContainerStarted","Data":"a1f4f7abfedf16564f2ed0f11a88468e991a75211e5f01a19a21353e138f69a1"} Jul 20 23:21:25.764210 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:25.764131 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" podStartSLOduration=1.7641105609999999 podStartE2EDuration="1.764110561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:21:25.761450101 +0000 UTC m=+1661.079802390" watchObservedRunningTime="2026-07-20 23:21:25.764110561 +0000 UTC m=+1661.082462851" Jul 20 23:21:25.969166 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:25.969134 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:26.974210 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:26.974180 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:27.163981 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.163945 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf"] Jul 20 23:21:27.164319 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.164306 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" Jul 20 23:21:27.164319 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.164321 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99cf074-b01f-4958-8721-bb2e35f692a5" containerName="istio-proxy" Jul 20 23:21:27.167476 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.167442 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.170141 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.170115 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-421a7b46-serving-cert\"" Jul 20 23:21:27.178487 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.178434 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf"] Jul 20 23:21:27.225969 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.225884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48918f87-b6f4-486c-a009-8fe50d749dd5-maas-api-tls\") pod \"maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.225969 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.225918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48918f87-b6f4-486c-a009-8fe50d749dd5-openshift-service-ca\") pod \"maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.225969 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.225952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24tsk\" (UniqueName: \"kubernetes.io/projected/48918f87-b6f4-486c-a009-8fe50d749dd5-kube-api-access-24tsk\") pod \"maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.326724 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.326687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48918f87-b6f4-486c-a009-8fe50d749dd5-openshift-service-ca\") pod \"maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.326906 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.326763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24tsk\" (UniqueName: \"kubernetes.io/projected/48918f87-b6f4-486c-a009-8fe50d749dd5-kube-api-access-24tsk\") pod \"maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.326906 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.326884 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48918f87-b6f4-486c-a009-8fe50d749dd5-maas-api-tls\") pod \"maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.327451 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.327428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48918f87-b6f4-486c-a009-8fe50d749dd5-openshift-service-ca\") pod \"maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.329299 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.329275 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48918f87-b6f4-486c-a009-8fe50d749dd5-maas-api-tls\") pod \"maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.335072 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.335054 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24tsk\" (UniqueName: \"kubernetes.io/projected/48918f87-b6f4-486c-a009-8fe50d749dd5-kube-api-access-24tsk\") pod \"maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.479635 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.479546 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:27.623581 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.623540 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf"] Jul 20 23:21:27.630372 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:21:27.630320 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48918f87_b6f4_486c_a009_8fe50d749dd5.slice/crio-bb404a01b26e32f2485a7a38a4c80f9a0bc4a32c9c05d66b3712aa939943e17c WatchSource:0}: Error finding container bb404a01b26e32f2485a7a38a4c80f9a0bc4a32c9c05d66b3712aa939943e17c: Status 404 returned error can't find the container with id bb404a01b26e32f2485a7a38a4c80f9a0bc4a32c9c05d66b3712aa939943e17c Jul 20 23:21:27.751390 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.751283 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" event={"ID":"48918f87-b6f4-486c-a009-8fe50d749dd5","Type":"ContainerStarted","Data":"bb404a01b26e32f2485a7a38a4c80f9a0bc4a32c9c05d66b3712aa939943e17c"} Jul 20 23:21:27.751619 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.751600 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:27.752541 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:27.752519 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:28.757854 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:28.757821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" event={"ID":"48918f87-b6f4-486c-a009-8fe50d749dd5","Type":"ContainerStarted","Data":"b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c"} Jul 20 23:21:28.758348 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:28.758042 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:28.775279 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:28.775228 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" podStartSLOduration=1.282794831 podStartE2EDuration="1.775213419s" totalImagesPullingTime="492.418588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:21:28.774301417 +0000 UTC m=+1664.092653726" watchObservedRunningTime="2026-07-20 23:21:28.775213419 +0000 UTC m=+1664.093565704" Jul 20 23:21:34.774826 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:34.774789 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:21:40.816170 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:40.816081 2574 generic.go:356] "Generic (PLEG): container finished" podID="116f786f-6f8f-4f82-b476-86993912e899" containerID="0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809" exitCode=0 Jul 20 23:21:40.816170 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:40.816154 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" event={"ID":"116f786f-6f8f-4f82-b476-86993912e899","Type":"ContainerDied","Data":"0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809"} Jul 20 23:21:40.816628 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:40.816557 2574 scope.go:117] "RemoveContainer" containerID="0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809" Jul 20 23:21:40.817613 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:40.817582 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:21:40.817734 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:40.817667 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:21:40.817734 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:40.817717 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:21:41.822806 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:41.822769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" event={"ID":"116f786f-6f8f-4f82-b476-86993912e899","Type":"ContainerStarted","Data":"b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966"} Jul 20 23:21:41.968930 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:41.968899 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:41.970663 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:41.970633 2574 patch_prober.go:28] interesting pod/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 20 23:21:41.970820 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:41.970703 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 20 23:21:42.752566 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:42.752512 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:42.970220 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:42.969731 2574 patch_prober.go:28] interesting pod/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 20 23:21:42.970220 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:42.969806 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 20 23:21:43.969334 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:43.969286 2574 patch_prober.go:28] interesting pod/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 20 23:21:43.969561 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:43.969361 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 20 23:21:44.969736 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:44.969695 2574 patch_prober.go:28] interesting pod/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 20 23:21:44.970316 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:44.970280 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 20 23:21:45.973079 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:45.973040 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:21:45.974049 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:21:45.974030 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:22:03.244315 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.244278 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf"] Jul 20 23:22:03.244781 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.244679 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" podUID="48918f87-b6f4-486c-a009-8fe50d749dd5" containerName="maas-api" containerID="cri-o://b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c" gracePeriod=30 Jul 20 23:22:03.501799 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.501730 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:22:03.546062 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.546016 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48918f87-b6f4-486c-a009-8fe50d749dd5-maas-api-tls\") pod \"48918f87-b6f4-486c-a009-8fe50d749dd5\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " Jul 20 23:22:03.546257 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.546074 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48918f87-b6f4-486c-a009-8fe50d749dd5-openshift-service-ca\") pod \"48918f87-b6f4-486c-a009-8fe50d749dd5\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " Jul 20 23:22:03.546257 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.546157 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24tsk\" (UniqueName: \"kubernetes.io/projected/48918f87-b6f4-486c-a009-8fe50d749dd5-kube-api-access-24tsk\") pod \"48918f87-b6f4-486c-a009-8fe50d749dd5\" (UID: \"48918f87-b6f4-486c-a009-8fe50d749dd5\") " Jul 20 23:22:03.546511 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.546481 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48918f87-b6f4-486c-a009-8fe50d749dd5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48918f87-b6f4-486c-a009-8fe50d749dd5" (UID: "48918f87-b6f4-486c-a009-8fe50d749dd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:22:03.548365 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.548341 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48918f87-b6f4-486c-a009-8fe50d749dd5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48918f87-b6f4-486c-a009-8fe50d749dd5" (UID: "48918f87-b6f4-486c-a009-8fe50d749dd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:22:03.548439 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.548364 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48918f87-b6f4-486c-a009-8fe50d749dd5-kube-api-access-24tsk" (OuterVolumeSpecName: "kube-api-access-24tsk") pod "48918f87-b6f4-486c-a009-8fe50d749dd5" (UID: "48918f87-b6f4-486c-a009-8fe50d749dd5"). InnerVolumeSpecName "kube-api-access-24tsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:22:03.647667 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.647628 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24tsk\" (UniqueName: \"kubernetes.io/projected/48918f87-b6f4-486c-a009-8fe50d749dd5-kube-api-access-24tsk\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:03.647667 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.647659 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48918f87-b6f4-486c-a009-8fe50d749dd5-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:03.647667 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.647669 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48918f87-b6f4-486c-a009-8fe50d749dd5-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:03.930077 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.930040 2574 generic.go:356] "Generic (PLEG): container finished" podID="48918f87-b6f4-486c-a009-8fe50d749dd5" containerID="b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c" exitCode=0 Jul 20 23:22:03.930077 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.930080 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" event={"ID":"48918f87-b6f4-486c-a009-8fe50d749dd5","Type":"ContainerDied","Data":"b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c"} Jul 20 23:22:03.930340 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.930091 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" Jul 20 23:22:03.930340 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.930122 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf" event={"ID":"48918f87-b6f4-486c-a009-8fe50d749dd5","Type":"ContainerDied","Data":"bb404a01b26e32f2485a7a38a4c80f9a0bc4a32c9c05d66b3712aa939943e17c"} Jul 20 23:22:03.930340 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.930144 2574 scope.go:117] "RemoveContainer" containerID="b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c" Jul 20 23:22:03.939909 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.939871 2574 scope.go:117] "RemoveContainer" containerID="b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c" Jul 20 23:22:03.945103 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:22:03.944868 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c\": container with ID starting with b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c not found: ID does not exist" containerID="b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c" Jul 20 23:22:03.945103 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.944915 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c"} err="failed to get container status \"b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c\": rpc error: code = NotFound desc = could not find container \"b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c\": container with ID starting with b0bddcab45f9640166d3dfe8f4b5674227a892a7a42b5f9e6ee3f1e457c96d0c not found: ID does not exist" Jul 20 23:22:03.953756 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.953725 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf"] Jul 20 23:22:03.958581 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:03.958545 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421a7b46-587d9b65b8-ppmlf"] Jul 20 23:22:05.228022 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:05.227986 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48918f87-b6f4-486c-a009-8fe50d749dd5" path="/var/lib/kubelet/pods/48918f87-b6f4-486c-a009-8fe50d749dd5/volumes" Jul 20 23:22:09.067601 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:09.067569 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp"] Jul 20 23:22:09.068059 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:09.067784 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" containerID="cri-o://b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966" gracePeriod=30 Jul 20 23:22:12.218166 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.218131 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7"] Jul 20 23:22:12.218731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.218534 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48918f87-b6f4-486c-a009-8fe50d749dd5" containerName="maas-api" Jul 20 23:22:12.218731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.218551 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48918f87-b6f4-486c-a009-8fe50d749dd5" containerName="maas-api" Jul 20 23:22:12.218731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.218631 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48918f87-b6f4-486c-a009-8fe50d749dd5" containerName="maas-api" Jul 20 23:22:12.222928 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.222907 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.225664 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.225641 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c1869ca5-serving-cert\"" Jul 20 23:22:12.233375 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.233348 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7"] Jul 20 23:22:12.326598 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.326563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd411d67-a02c-431a-bb1d-4a0307ed0b76-maas-api-tls\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.326598 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.326603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mtsf\" (UniqueName: \"kubernetes.io/projected/cd411d67-a02c-431a-bb1d-4a0307ed0b76-kube-api-access-6mtsf\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.326872 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.326737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd411d67-a02c-431a-bb1d-4a0307ed0b76-openshift-service-ca\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.427785 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.427750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd411d67-a02c-431a-bb1d-4a0307ed0b76-openshift-service-ca\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.428007 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.427842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd411d67-a02c-431a-bb1d-4a0307ed0b76-maas-api-tls\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.428007 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.427868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mtsf\" (UniqueName: \"kubernetes.io/projected/cd411d67-a02c-431a-bb1d-4a0307ed0b76-kube-api-access-6mtsf\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.428131 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:22:12.428004 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-serving-cert: secret "maas-api-e2e-derive-c1869ca5-serving-cert" not found Jul 20 23:22:12.428131 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:22:12.428093 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd411d67-a02c-431a-bb1d-4a0307ed0b76-maas-api-tls podName:cd411d67-a02c-431a-bb1d-4a0307ed0b76 nodeName:}" failed. No retries permitted until 2026-07-20 23:22:12.92806681 +0000 UTC m=+1708.246419089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd411d67-a02c-431a-bb1d-4a0307ed0b76-maas-api-tls") pod "maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" (UID: "cd411d67-a02c-431a-bb1d-4a0307ed0b76") : secret "maas-api-e2e-derive-c1869ca5-serving-cert" not found Jul 20 23:22:12.428517 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.428496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd411d67-a02c-431a-bb1d-4a0307ed0b76-openshift-service-ca\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.439408 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.439377 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mtsf\" (UniqueName: \"kubernetes.io/projected/cd411d67-a02c-431a-bb1d-4a0307ed0b76-kube-api-access-6mtsf\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.932647 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.932611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd411d67-a02c-431a-bb1d-4a0307ed0b76-maas-api-tls\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:12.935041 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:12.935014 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd411d67-a02c-431a-bb1d-4a0307ed0b76-maas-api-tls\") pod \"maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:13.134867 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:13.134828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:13.274079 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:13.274053 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7"] Jul 20 23:22:13.276270 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:22:13.276239 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd411d67_a02c_431a_bb1d_4a0307ed0b76.slice/crio-d9dd0923e777233188fb9685652b2a2ae3cfe43542e4e395e81ef55ab47a095a WatchSource:0}: Error finding container d9dd0923e777233188fb9685652b2a2ae3cfe43542e4e395e81ef55ab47a095a: Status 404 returned error can't find the container with id d9dd0923e777233188fb9685652b2a2ae3cfe43542e4e395e81ef55ab47a095a Jul 20 23:22:13.981200 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:13.981112 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" event={"ID":"cd411d67-a02c-431a-bb1d-4a0307ed0b76","Type":"ContainerStarted","Data":"2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07"} Jul 20 23:22:13.981200 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:13.981165 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" event={"ID":"cd411d67-a02c-431a-bb1d-4a0307ed0b76","Type":"ContainerStarted","Data":"d9dd0923e777233188fb9685652b2a2ae3cfe43542e4e395e81ef55ab47a095a"} Jul 20 23:22:13.981392 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:13.981246 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:13.999812 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:13.999764 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" podStartSLOduration=1.607115262 podStartE2EDuration="1.999747436s" totalImagesPullingTime="392.632174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:22:13.997394926 +0000 UTC m=+1709.315747211" watchObservedRunningTime="2026-07-20 23:22:13.999747436 +0000 UTC m=+1709.318099721" Jul 20 23:22:14.331729 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.331706 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:22:14.446736 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.446702 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116f786f-6f8f-4f82-b476-86993912e899-istio-podinfo\") pod \"116f786f-6f8f-4f82-b476-86993912e899\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " Jul 20 23:22:14.446920 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.446767 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-credential-socket\") pod \"116f786f-6f8f-4f82-b476-86993912e899\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " Jul 20 23:22:14.446920 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.446828 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-certs\") pod \"116f786f-6f8f-4f82-b476-86993912e899\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " Jul 20 23:22:14.446920 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.446863 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-istio-token\") pod \"116f786f-6f8f-4f82-b476-86993912e899\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " Jul 20 23:22:14.446920 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.446896 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-socket\") pod \"116f786f-6f8f-4f82-b476-86993912e899\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " Jul 20 23:22:14.447302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.446923 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116f786f-6f8f-4f82-b476-86993912e899-istiod-ca-cert\") pod \"116f786f-6f8f-4f82-b476-86993912e899\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " Jul 20 23:22:14.447302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.446947 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-data\") pod \"116f786f-6f8f-4f82-b476-86993912e899\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " Jul 20 23:22:14.447302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.446980 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-envoy\") pod \"116f786f-6f8f-4f82-b476-86993912e899\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " Jul 20 23:22:14.447302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.447016 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n22gh\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-kube-api-access-n22gh\") pod \"116f786f-6f8f-4f82-b476-86993912e899\" (UID: \"116f786f-6f8f-4f82-b476-86993912e899\") " Jul 20 23:22:14.447302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.447104 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "116f786f-6f8f-4f82-b476-86993912e899" (UID: "116f786f-6f8f-4f82-b476-86993912e899"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:22:14.447302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.447120 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "116f786f-6f8f-4f82-b476-86993912e899" (UID: "116f786f-6f8f-4f82-b476-86993912e899"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:22:14.447302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.447193 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "116f786f-6f8f-4f82-b476-86993912e899" (UID: "116f786f-6f8f-4f82-b476-86993912e899"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:22:14.447726 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.447328 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-credential-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:14.447726 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.447355 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:14.447726 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.447358 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116f786f-6f8f-4f82-b476-86993912e899-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "116f786f-6f8f-4f82-b476-86993912e899" (UID: "116f786f-6f8f-4f82-b476-86993912e899"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:22:14.447726 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.447371 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-workload-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:14.447925 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.447903 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-data" (OuterVolumeSpecName: "istio-data") pod "116f786f-6f8f-4f82-b476-86993912e899" (UID: "116f786f-6f8f-4f82-b476-86993912e899"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:22:14.449265 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.449236 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-istio-token" (OuterVolumeSpecName: "istio-token") pod "116f786f-6f8f-4f82-b476-86993912e899" (UID: "116f786f-6f8f-4f82-b476-86993912e899"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:22:14.449384 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.449350 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "116f786f-6f8f-4f82-b476-86993912e899" (UID: "116f786f-6f8f-4f82-b476-86993912e899"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:22:14.449481 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.449437 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-kube-api-access-n22gh" (OuterVolumeSpecName: "kube-api-access-n22gh") pod "116f786f-6f8f-4f82-b476-86993912e899" (UID: "116f786f-6f8f-4f82-b476-86993912e899"). InnerVolumeSpecName "kube-api-access-n22gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:22:14.449554 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.449435 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/116f786f-6f8f-4f82-b476-86993912e899-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "116f786f-6f8f-4f82-b476-86993912e899" (UID: "116f786f-6f8f-4f82-b476-86993912e899"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 23:22:14.547881 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.547783 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116f786f-6f8f-4f82-b476-86993912e899-istio-podinfo\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:14.547881 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.547825 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-istio-token\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:14.547881 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.547836 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116f786f-6f8f-4f82-b476-86993912e899-istiod-ca-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:14.547881 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.547844 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-data\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:14.547881 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.547854 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116f786f-6f8f-4f82-b476-86993912e899-istio-envoy\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:14.547881 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.547862 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n22gh\" (UniqueName: \"kubernetes.io/projected/116f786f-6f8f-4f82-b476-86993912e899-kube-api-access-n22gh\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:14.987166 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.987127 2574 generic.go:356] "Generic (PLEG): container finished" podID="116f786f-6f8f-4f82-b476-86993912e899" containerID="b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966" exitCode=0 Jul 20 23:22:14.987378 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.987181 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" Jul 20 23:22:14.987378 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.987212 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" event={"ID":"116f786f-6f8f-4f82-b476-86993912e899","Type":"ContainerDied","Data":"b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966"} Jul 20 23:22:14.987378 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.987274 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp" event={"ID":"116f786f-6f8f-4f82-b476-86993912e899","Type":"ContainerDied","Data":"a1f4f7abfedf16564f2ed0f11a88468e991a75211e5f01a19a21353e138f69a1"} Jul 20 23:22:14.987378 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:14.987304 2574 scope.go:117] "RemoveContainer" containerID="b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966" Jul 20 23:22:15.000078 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:15.000058 2574 scope.go:117] "RemoveContainer" containerID="0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809" Jul 20 23:22:15.008023 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:15.008004 2574 scope.go:117] "RemoveContainer" containerID="b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966" Jul 20 23:22:15.008299 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:22:15.008276 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966\": container with ID starting with b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966 not found: ID does not exist" containerID="b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966" Jul 20 23:22:15.008362 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:15.008306 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966"} err="failed to get container status \"b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966\": rpc error: code = NotFound desc = could not find container \"b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966\": container with ID starting with b2e36041180a2ecd91990ef44fdcab0c57a2a071102c46923b1487bff2977966 not found: ID does not exist" Jul 20 23:22:15.008362 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:15.008324 2574 scope.go:117] "RemoveContainer" containerID="0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809" Jul 20 23:22:15.008576 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:22:15.008558 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809\": container with ID starting with 0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809 not found: ID does not exist" containerID="0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809" Jul 20 23:22:15.008621 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:15.008579 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809"} err="failed to get container status \"0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809\": rpc error: code = NotFound desc = could not find container \"0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809\": container with ID starting with 0ccd16bc3ca4ceb2183a12a99934ba1209a8d32ac06713f8061e895d41e28809 not found: ID does not exist" Jul 20 23:22:15.014287 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:15.014265 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp"] Jul 20 23:22:15.017829 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:15.017808 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-421a7b46-openshift-default-85ffffcd9-vx5xp"] Jul 20 23:22:15.229258 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:15.229224 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="116f786f-6f8f-4f82-b476-86993912e899" path="/var/lib/kubelet/pods/116f786f-6f8f-4f82-b476-86993912e899/volumes" Jul 20 23:22:19.993082 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:19.993047 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:47.330909 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.330862 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7"] Jul 20 23:22:47.331417 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.331112 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" podUID="cd411d67-a02c-431a-bb1d-4a0307ed0b76" containerName="maas-api" containerID="cri-o://2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07" gracePeriod=30 Jul 20 23:22:47.599951 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.599926 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:47.646212 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.646177 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd411d67-a02c-431a-bb1d-4a0307ed0b76-maas-api-tls\") pod \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " Jul 20 23:22:47.646399 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.646294 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd411d67-a02c-431a-bb1d-4a0307ed0b76-openshift-service-ca\") pod \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " Jul 20 23:22:47.646399 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.646320 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mtsf\" (UniqueName: \"kubernetes.io/projected/cd411d67-a02c-431a-bb1d-4a0307ed0b76-kube-api-access-6mtsf\") pod \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\" (UID: \"cd411d67-a02c-431a-bb1d-4a0307ed0b76\") " Jul 20 23:22:47.646683 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.646657 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd411d67-a02c-431a-bb1d-4a0307ed0b76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd411d67-a02c-431a-bb1d-4a0307ed0b76" (UID: "cd411d67-a02c-431a-bb1d-4a0307ed0b76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:22:47.648354 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.648327 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd411d67-a02c-431a-bb1d-4a0307ed0b76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd411d67-a02c-431a-bb1d-4a0307ed0b76" (UID: "cd411d67-a02c-431a-bb1d-4a0307ed0b76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:22:47.648475 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.648377 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd411d67-a02c-431a-bb1d-4a0307ed0b76-kube-api-access-6mtsf" (OuterVolumeSpecName: "kube-api-access-6mtsf") pod "cd411d67-a02c-431a-bb1d-4a0307ed0b76" (UID: "cd411d67-a02c-431a-bb1d-4a0307ed0b76"). InnerVolumeSpecName "kube-api-access-6mtsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:22:47.747992 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.747954 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd411d67-a02c-431a-bb1d-4a0307ed0b76-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:47.747992 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.747985 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mtsf\" (UniqueName: \"kubernetes.io/projected/cd411d67-a02c-431a-bb1d-4a0307ed0b76-kube-api-access-6mtsf\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:47.747992 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:47.747995 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd411d67-a02c-431a-bb1d-4a0307ed0b76-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:22:48.143222 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:48.143188 2574 generic.go:356] "Generic (PLEG): container finished" podID="cd411d67-a02c-431a-bb1d-4a0307ed0b76" containerID="2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07" exitCode=0 Jul 20 23:22:48.143430 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:48.143229 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" Jul 20 23:22:48.143430 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:48.143275 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" event={"ID":"cd411d67-a02c-431a-bb1d-4a0307ed0b76","Type":"ContainerDied","Data":"2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07"} Jul 20 23:22:48.143430 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:48.143328 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7" event={"ID":"cd411d67-a02c-431a-bb1d-4a0307ed0b76","Type":"ContainerDied","Data":"d9dd0923e777233188fb9685652b2a2ae3cfe43542e4e395e81ef55ab47a095a"} Jul 20 23:22:48.143430 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:48.143360 2574 scope.go:117] "RemoveContainer" containerID="2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07" Jul 20 23:22:48.157731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:48.157703 2574 scope.go:117] "RemoveContainer" containerID="2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07" Jul 20 23:22:48.158034 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:22:48.158016 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07\": container with ID starting with 2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07 not found: ID does not exist" containerID="2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07" Jul 20 23:22:48.158106 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:48.158040 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07"} err="failed to get container status \"2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07\": rpc error: code = NotFound desc = could not find container \"2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07\": container with ID starting with 2b203c423aca9c64ea705a47b94634eb70245ac78727a395cea42dee89a2af07 not found: ID does not exist" Jul 20 23:22:48.168644 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:48.168618 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7"] Jul 20 23:22:48.172255 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:48.172231 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c1869ca5-d5cb849fb-dvxv7"] Jul 20 23:22:49.226814 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:22:49.226783 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd411d67-a02c-431a-bb1d-4a0307ed0b76" path="/var/lib/kubelet/pods/cd411d67-a02c-431a-bb1d-4a0307ed0b76/volumes" Jul 20 23:23:45.181306 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:23:45.181278 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:23:45.182669 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:23:45.182648 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:24:29.032435 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.032395 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q"] Jul 20 23:24:29.033043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.032841 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" Jul 20 23:24:29.033043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.032855 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" Jul 20 23:24:29.033043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.032869 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" Jul 20 23:24:29.033043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.032876 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" Jul 20 23:24:29.033043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.032885 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd411d67-a02c-431a-bb1d-4a0307ed0b76" containerName="maas-api" Jul 20 23:24:29.033043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.032890 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd411d67-a02c-431a-bb1d-4a0307ed0b76" containerName="maas-api" Jul 20 23:24:29.033043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.032943 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" Jul 20 23:24:29.033043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.032952 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd411d67-a02c-431a-bb1d-4a0307ed0b76" containerName="maas-api" Jul 20 23:24:29.036011 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.035988 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.041398 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.041375 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e0e09171-openshift-default-dockercfg-fsbfd\"" Jul 20 23:24:29.061437 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.061401 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q"] Jul 20 23:24:29.130122 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.130084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-token\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.130122 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.130123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-envoy\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.130342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.130147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-data\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.130342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.130218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-socket\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.130342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.130252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-credential-socket\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.130342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.130282 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87153198-8a0f-439b-88a2-47dc53db5c0f-istiod-ca-cert\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.130342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.130299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wkfz\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-kube-api-access-9wkfz\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.130342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.130342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-certs\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.130581 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.130357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-podinfo\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.230975 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.230938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-podinfo\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231175 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-token\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231175 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-envoy\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231175 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-data\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231175 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-socket\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231175 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-credential-socket\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231454 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87153198-8a0f-439b-88a2-47dc53db5c0f-istiod-ca-cert\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231454 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wkfz\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-kube-api-access-9wkfz\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231454 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-certs\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231801 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231775 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-certs\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.231999 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231902 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-socket\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.232126 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.231936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-data\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.232248 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.232182 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-credential-socket\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.232568 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.232542 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87153198-8a0f-439b-88a2-47dc53db5c0f-istiod-ca-cert\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.233995 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.233969 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-envoy\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.234094 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.234074 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-podinfo\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.241081 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.241029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-token\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.241237 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.241212 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wkfz\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-kube-api-access-9wkfz\") pod \"e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.347006 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.346909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:29.500868 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.500835 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q"] Jul 20 23:24:29.503722 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:24:29.503677 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87153198_8a0f_439b_88a2_47dc53db5c0f.slice/crio-0e3d8d7a171f5880f6ea66ab8520bf4ec6f9079a0d4b3ac7b01aea6b66cd7c1b WatchSource:0}: Error finding container 0e3d8d7a171f5880f6ea66ab8520bf4ec6f9079a0d4b3ac7b01aea6b66cd7c1b: Status 404 returned error can't find the container with id 0e3d8d7a171f5880f6ea66ab8520bf4ec6f9079a0d4b3ac7b01aea6b66cd7c1b Jul 20 23:24:29.506543 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.506425 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:24:29.506543 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.506522 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:24:29.506725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.506565 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:24:29.594178 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.594141 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" event={"ID":"87153198-8a0f-439b-88a2-47dc53db5c0f","Type":"ContainerStarted","Data":"b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9"} Jul 20 23:24:29.594365 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.594208 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" event={"ID":"87153198-8a0f-439b-88a2-47dc53db5c0f","Type":"ContainerStarted","Data":"0e3d8d7a171f5880f6ea66ab8520bf4ec6f9079a0d4b3ac7b01aea6b66cd7c1b"} Jul 20 23:24:29.614875 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:29.614814 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" podStartSLOduration=0.614792675 podStartE2EDuration="614.792675ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:24:29.613292145 +0000 UTC m=+1844.931644432" watchObservedRunningTime="2026-07-20 23:24:29.614792675 +0000 UTC m=+1844.933144962" Jul 20 23:24:30.347633 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:30.347597 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:30.353529 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:30.353502 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:30.599510 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:30.599476 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:30.600434 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:30.600416 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:31.941575 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:31.941543 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h"] Jul 20 23:24:31.942300 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:31.942267 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="116f786f-6f8f-4f82-b476-86993912e899" containerName="istio-proxy" Jul 20 23:24:31.945717 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:31.945696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:31.948512 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:31.948485 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e0e09171-serving-cert\"" Jul 20 23:24:31.954933 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:31.954905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8ae186-c049-4b39-8a3c-b82dd60c379c-maas-api-tls\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:31.955066 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:31.955013 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqvsm\" (UniqueName: \"kubernetes.io/projected/fe8ae186-c049-4b39-8a3c-b82dd60c379c-kube-api-access-xqvsm\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:31.955066 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:31.955053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ae186-c049-4b39-8a3c-b82dd60c379c-openshift-service-ca\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:31.957266 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:31.956994 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h"] Jul 20 23:24:32.055773 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:32.055730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqvsm\" (UniqueName: \"kubernetes.io/projected/fe8ae186-c049-4b39-8a3c-b82dd60c379c-kube-api-access-xqvsm\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:32.055942 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:32.055787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ae186-c049-4b39-8a3c-b82dd60c379c-openshift-service-ca\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:32.055942 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:32.055881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8ae186-c049-4b39-8a3c-b82dd60c379c-maas-api-tls\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:32.056076 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:24:32.056021 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-serving-cert: secret "maas-api-e2e-mt-e0e09171-serving-cert" not found Jul 20 23:24:32.056140 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:24:32.056097 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe8ae186-c049-4b39-8a3c-b82dd60c379c-maas-api-tls podName:fe8ae186-c049-4b39-8a3c-b82dd60c379c nodeName:}" failed. No retries permitted until 2026-07-20 23:24:32.556073037 +0000 UTC m=+1847.874425318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe8ae186-c049-4b39-8a3c-b82dd60c379c-maas-api-tls") pod "maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" (UID: "fe8ae186-c049-4b39-8a3c-b82dd60c379c") : secret "maas-api-e2e-mt-e0e09171-serving-cert" not found Jul 20 23:24:32.056536 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:32.056513 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ae186-c049-4b39-8a3c-b82dd60c379c-openshift-service-ca\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:32.068849 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:32.068819 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqvsm\" (UniqueName: \"kubernetes.io/projected/fe8ae186-c049-4b39-8a3c-b82dd60c379c-kube-api-access-xqvsm\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:32.559311 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:32.559275 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8ae186-c049-4b39-8a3c-b82dd60c379c-maas-api-tls\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:32.562096 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:32.562063 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8ae186-c049-4b39-8a3c-b82dd60c379c-maas-api-tls\") pod \"maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:32.858262 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:32.858222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:32.997569 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:32.997541 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h"] Jul 20 23:24:32.999486 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:24:32.999433 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8ae186_c049_4b39_8a3c_b82dd60c379c.slice/crio-d6dd4c9ea9f4d3c611fbbf61fc77f96190d99f2f93794c2597a5c2259d9d8af2 WatchSource:0}: Error finding container d6dd4c9ea9f4d3c611fbbf61fc77f96190d99f2f93794c2597a5c2259d9d8af2: Status 404 returned error can't find the container with id d6dd4c9ea9f4d3c611fbbf61fc77f96190d99f2f93794c2597a5c2259d9d8af2 Jul 20 23:24:33.619160 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:33.618751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" event={"ID":"fe8ae186-c049-4b39-8a3c-b82dd60c379c","Type":"ContainerStarted","Data":"d6dd4c9ea9f4d3c611fbbf61fc77f96190d99f2f93794c2597a5c2259d9d8af2"} Jul 20 23:24:34.625509 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:34.625476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" event={"ID":"fe8ae186-c049-4b39-8a3c-b82dd60c379c","Type":"ContainerStarted","Data":"480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc"} Jul 20 23:24:34.625932 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:34.625579 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:34.641207 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:34.641159 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" podStartSLOduration=3.057473307 podStartE2EDuration="3.64114367s" totalImagesPullingTime="583.670363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:24:34.640766004 +0000 UTC m=+1849.959118291" watchObservedRunningTime="2026-07-20 23:24:34.64114367 +0000 UTC m=+1849.959495950" Jul 20 23:24:40.635661 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:40.635627 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:43.670349 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:43.670311 2574 generic.go:356] "Generic (PLEG): container finished" podID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerID="b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9" exitCode=0 Jul 20 23:24:43.670840 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:43.670384 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" event={"ID":"87153198-8a0f-439b-88a2-47dc53db5c0f","Type":"ContainerDied","Data":"b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9"} Jul 20 23:24:43.670840 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:43.670794 2574 scope.go:117] "RemoveContainer" containerID="b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9" Jul 20 23:24:43.671823 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:43.671794 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:24:43.671944 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:43.671881 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:24:43.672003 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:43.671947 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:24:44.676626 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:44.676593 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" event={"ID":"87153198-8a0f-439b-88a2-47dc53db5c0f","Type":"ContainerStarted","Data":"d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a"} Jul 20 23:24:45.347351 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:45.347305 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:45.348413 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:45.348386 2574 patch_prober.go:28] interesting pod/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 20 23:24:45.348530 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:45.348454 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 20 23:24:45.600210 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:45.600182 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:46.347612 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:46.347576 2574 patch_prober.go:28] interesting pod/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 20 23:24:46.348000 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:46.347629 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 20 23:24:47.347807 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:47.347770 2574 patch_prober.go:28] interesting pod/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 20 23:24:47.348205 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:47.347824 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 20 23:24:48.347571 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:48.347535 2574 patch_prober.go:28] interesting pod/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 20 23:24:48.347761 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:48.347606 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 20 23:24:49.351623 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:49.351593 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:49.352577 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:49.352558 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:24:53.942217 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:53.942185 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h"] Jul 20 23:24:53.942713 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:53.942434 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" podUID="fe8ae186-c049-4b39-8a3c-b82dd60c379c" containerName="maas-api" containerID="cri-o://480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc" gracePeriod=30 Jul 20 23:24:54.193762 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.193703 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:54.256510 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.256478 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ae186-c049-4b39-8a3c-b82dd60c379c-openshift-service-ca\") pod \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " Jul 20 23:24:54.256679 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.256515 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqvsm\" (UniqueName: \"kubernetes.io/projected/fe8ae186-c049-4b39-8a3c-b82dd60c379c-kube-api-access-xqvsm\") pod \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " Jul 20 23:24:54.256679 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.256569 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8ae186-c049-4b39-8a3c-b82dd60c379c-maas-api-tls\") pod \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\" (UID: \"fe8ae186-c049-4b39-8a3c-b82dd60c379c\") " Jul 20 23:24:54.256924 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.256895 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8ae186-c049-4b39-8a3c-b82dd60c379c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe8ae186-c049-4b39-8a3c-b82dd60c379c" (UID: "fe8ae186-c049-4b39-8a3c-b82dd60c379c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:24:54.258693 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.258656 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8ae186-c049-4b39-8a3c-b82dd60c379c-kube-api-access-xqvsm" (OuterVolumeSpecName: "kube-api-access-xqvsm") pod "fe8ae186-c049-4b39-8a3c-b82dd60c379c" (UID: "fe8ae186-c049-4b39-8a3c-b82dd60c379c"). InnerVolumeSpecName "kube-api-access-xqvsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:24:54.258935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.258916 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8ae186-c049-4b39-8a3c-b82dd60c379c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe8ae186-c049-4b39-8a3c-b82dd60c379c" (UID: "fe8ae186-c049-4b39-8a3c-b82dd60c379c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:24:54.357375 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.357339 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ae186-c049-4b39-8a3c-b82dd60c379c-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:24:54.357375 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.357370 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqvsm\" (UniqueName: \"kubernetes.io/projected/fe8ae186-c049-4b39-8a3c-b82dd60c379c-kube-api-access-xqvsm\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:24:54.357375 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.357381 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8ae186-c049-4b39-8a3c-b82dd60c379c-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:24:54.728908 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.728875 2574 generic.go:356] "Generic (PLEG): container finished" podID="fe8ae186-c049-4b39-8a3c-b82dd60c379c" containerID="480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc" exitCode=0 Jul 20 23:24:54.729079 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.728920 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" Jul 20 23:24:54.729079 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.728938 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" event={"ID":"fe8ae186-c049-4b39-8a3c-b82dd60c379c","Type":"ContainerDied","Data":"480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc"} Jul 20 23:24:54.729079 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.728980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h" event={"ID":"fe8ae186-c049-4b39-8a3c-b82dd60c379c","Type":"ContainerDied","Data":"d6dd4c9ea9f4d3c611fbbf61fc77f96190d99f2f93794c2597a5c2259d9d8af2"} Jul 20 23:24:54.729079 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.729002 2574 scope.go:117] "RemoveContainer" containerID="480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc" Jul 20 23:24:54.738314 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.738295 2574 scope.go:117] "RemoveContainer" containerID="480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc" Jul 20 23:24:54.738601 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:24:54.738582 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc\": container with ID starting with 480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc not found: ID does not exist" containerID="480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc" Jul 20 23:24:54.738679 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.738613 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc"} err="failed to get container status \"480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc\": rpc error: code = NotFound desc = could not find container \"480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc\": container with ID starting with 480e8bb03fa9ae6a745daf385713221bc332ff6f71d94feb0c86d7394431c9fc not found: ID does not exist" Jul 20 23:24:54.750884 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.750834 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h"] Jul 20 23:24:54.752604 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:54.752581 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0e09171-677d7648b4-7lf5h"] Jul 20 23:24:55.227974 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:24:55.227934 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8ae186-c049-4b39-8a3c-b82dd60c379c" path="/var/lib/kubelet/pods/fe8ae186-c049-4b39-8a3c-b82dd60c379c/volumes" Jul 20 23:25:04.701566 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:04.701531 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q"] Jul 20 23:25:04.702133 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:04.701727 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" containerID="cri-o://d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a" gracePeriod=30 Jul 20 23:25:09.958634 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:09.958607 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:25:10.090188 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090088 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-credential-socket\") pod \"87153198-8a0f-439b-88a2-47dc53db5c0f\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " Jul 20 23:25:10.090188 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090152 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-certs\") pod \"87153198-8a0f-439b-88a2-47dc53db5c0f\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " Jul 20 23:25:10.090419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090201 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-podinfo\") pod \"87153198-8a0f-439b-88a2-47dc53db5c0f\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " Jul 20 23:25:10.090419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090237 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-socket\") pod \"87153198-8a0f-439b-88a2-47dc53db5c0f\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " Jul 20 23:25:10.090419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090261 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-token\") pod \"87153198-8a0f-439b-88a2-47dc53db5c0f\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " Jul 20 23:25:10.090419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090281 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-envoy\") pod \"87153198-8a0f-439b-88a2-47dc53db5c0f\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " Jul 20 23:25:10.090419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090316 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-data\") pod \"87153198-8a0f-439b-88a2-47dc53db5c0f\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " Jul 20 23:25:10.090419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090337 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wkfz\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-kube-api-access-9wkfz\") pod \"87153198-8a0f-439b-88a2-47dc53db5c0f\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " Jul 20 23:25:10.090419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090393 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87153198-8a0f-439b-88a2-47dc53db5c0f" (UID: "87153198-8a0f-439b-88a2-47dc53db5c0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:25:10.090419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090412 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87153198-8a0f-439b-88a2-47dc53db5c0f-istiod-ca-cert\") pod \"87153198-8a0f-439b-88a2-47dc53db5c0f\" (UID: \"87153198-8a0f-439b-88a2-47dc53db5c0f\") " Jul 20 23:25:10.090868 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090524 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87153198-8a0f-439b-88a2-47dc53db5c0f" (UID: "87153198-8a0f-439b-88a2-47dc53db5c0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:25:10.090868 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090624 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87153198-8a0f-439b-88a2-47dc53db5c0f" (UID: "87153198-8a0f-439b-88a2-47dc53db5c0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:25:10.090868 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090766 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-credential-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:25:10.090868 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090786 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:25:10.090868 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090807 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-workload-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:25:10.091129 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.090878 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87153198-8a0f-439b-88a2-47dc53db5c0f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87153198-8a0f-439b-88a2-47dc53db5c0f" (UID: "87153198-8a0f-439b-88a2-47dc53db5c0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:25:10.091620 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.091584 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-data" (OuterVolumeSpecName: "istio-data") pod "87153198-8a0f-439b-88a2-47dc53db5c0f" (UID: "87153198-8a0f-439b-88a2-47dc53db5c0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:25:10.092617 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.092589 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87153198-8a0f-439b-88a2-47dc53db5c0f" (UID: "87153198-8a0f-439b-88a2-47dc53db5c0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 23:25:10.092617 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.092595 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87153198-8a0f-439b-88a2-47dc53db5c0f" (UID: "87153198-8a0f-439b-88a2-47dc53db5c0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:25:10.092765 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.092618 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-token" (OuterVolumeSpecName: "istio-token") pod "87153198-8a0f-439b-88a2-47dc53db5c0f" (UID: "87153198-8a0f-439b-88a2-47dc53db5c0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:25:10.092839 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.092817 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-kube-api-access-9wkfz" (OuterVolumeSpecName: "kube-api-access-9wkfz") pod "87153198-8a0f-439b-88a2-47dc53db5c0f" (UID: "87153198-8a0f-439b-88a2-47dc53db5c0f"). InnerVolumeSpecName "kube-api-access-9wkfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:25:10.191770 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.191732 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87153198-8a0f-439b-88a2-47dc53db5c0f-istiod-ca-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:25:10.191770 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.191766 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-podinfo\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:25:10.191770 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.191778 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-token\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:25:10.192084 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.191789 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-envoy\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:25:10.192084 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.191801 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87153198-8a0f-439b-88a2-47dc53db5c0f-istio-data\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:25:10.192084 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.191811 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wkfz\" (UniqueName: \"kubernetes.io/projected/87153198-8a0f-439b-88a2-47dc53db5c0f-kube-api-access-9wkfz\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:25:10.809897 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.809862 2574 generic.go:356] "Generic (PLEG): container finished" podID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerID="d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a" exitCode=0 Jul 20 23:25:10.810078 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.809909 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" Jul 20 23:25:10.810078 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.810019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" event={"ID":"87153198-8a0f-439b-88a2-47dc53db5c0f","Type":"ContainerDied","Data":"d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a"} Jul 20 23:25:10.810179 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.810083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q" event={"ID":"87153198-8a0f-439b-88a2-47dc53db5c0f","Type":"ContainerDied","Data":"0e3d8d7a171f5880f6ea66ab8520bf4ec6f9079a0d4b3ac7b01aea6b66cd7c1b"} Jul 20 23:25:10.810179 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.810119 2574 scope.go:117] "RemoveContainer" containerID="d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a" Jul 20 23:25:10.820106 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.819955 2574 scope.go:117] "RemoveContainer" containerID="b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9" Jul 20 23:25:10.828178 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.828159 2574 scope.go:117] "RemoveContainer" containerID="d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a" Jul 20 23:25:10.828437 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:25:10.828418 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a\": container with ID starting with d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a not found: ID does not exist" containerID="d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a" Jul 20 23:25:10.828511 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.828446 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a"} err="failed to get container status \"d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a\": rpc error: code = NotFound desc = could not find container \"d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a\": container with ID starting with d356744f684c6885b8287127306b7de842e568797e504de54e36f88e04bf498a not found: ID does not exist" Jul 20 23:25:10.828511 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.828481 2574 scope.go:117] "RemoveContainer" containerID="b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9" Jul 20 23:25:10.828723 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:25:10.828708 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9\": container with ID starting with b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9 not found: ID does not exist" containerID="b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9" Jul 20 23:25:10.828777 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.828726 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9"} err="failed to get container status \"b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9\": rpc error: code = NotFound desc = could not find container \"b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9\": container with ID starting with b8f1d7bc28407a3379a2f91a3bf9970fd751ab226c53dfa8d6af7554b35f1cf9 not found: ID does not exist" Jul 20 23:25:10.833826 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.833800 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q"] Jul 20 23:25:10.837707 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:10.837681 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e0e09171-openshift-default-6d8c5b9fff-29n7q"] Jul 20 23:25:11.227862 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:11.227832 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" path="/var/lib/kubelet/pods/87153198-8a0f-439b-88a2-47dc53db5c0f/volumes" Jul 20 23:25:24.080311 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.080277 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh"] Jul 20 23:25:24.080852 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.080835 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" Jul 20 23:25:24.080899 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.080856 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" Jul 20 23:25:24.080899 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.080876 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" Jul 20 23:25:24.080899 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.080885 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" Jul 20 23:25:24.081004 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.080903 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8ae186-c049-4b39-8a3c-b82dd60c379c" containerName="maas-api" Jul 20 23:25:24.081004 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.080913 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8ae186-c049-4b39-8a3c-b82dd60c379c" containerName="maas-api" Jul 20 23:25:24.081004 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.080990 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8ae186-c049-4b39-8a3c-b82dd60c379c" containerName="maas-api" Jul 20 23:25:24.081004 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.081002 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" Jul 20 23:25:24.084358 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.084337 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.087112 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.087087 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2db333cd-openshift-default-dockercfg-9tss9\"" Jul 20 23:25:24.095274 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.095247 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh"] Jul 20 23:25:24.117088 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.117052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-token\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.117271 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.117106 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-socket\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.117271 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.117187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qhd\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-kube-api-access-b8qhd\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.117271 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.117239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-credential-socket\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.117532 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.117297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-envoy\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.117532 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.117346 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-data\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.117532 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.117371 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istiod-ca-cert\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.117766 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.117606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-certs\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.117766 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.117644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-podinfo\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.218421 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.218385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-token\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.218643 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.218435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-socket\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.218643 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.218501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qhd\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-kube-api-access-b8qhd\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.218643 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.218523 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-credential-socket\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.218643 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.218556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-envoy\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.218643 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.218591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-data\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.218643 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.218616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istiod-ca-cert\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.218932 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.218659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-certs\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.218932 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.218682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-podinfo\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.219078 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.219047 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-socket\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.219275 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.219133 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-credential-socket\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.219275 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.219187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-data\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.219428 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.219198 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-certs\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.219516 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.219492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istiod-ca-cert\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.221037 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.221019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-envoy\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.221259 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.221240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-podinfo\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.226300 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.226274 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-token\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.226564 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.226546 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qhd\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-kube-api-access-b8qhd\") pod \"e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.394994 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.394957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:24.531027 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.531001 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh"] Jul 20 23:25:24.532844 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:25:24.532815 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cd07c0_db4a_4ca2_b045_01446e4b6e53.slice/crio-72c797b59cb72748802330cb8f7ca494abc52aaa9b4eb7390a7e0284ee609db3 WatchSource:0}: Error finding container 72c797b59cb72748802330cb8f7ca494abc52aaa9b4eb7390a7e0284ee609db3: Status 404 returned error can't find the container with id 72c797b59cb72748802330cb8f7ca494abc52aaa9b4eb7390a7e0284ee609db3 Jul 20 23:25:24.534896 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.534861 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:25:24.534976 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.534945 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:25:24.535014 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.534980 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:25:24.875538 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.875494 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" event={"ID":"44cd07c0-db4a-4ca2-b045-01446e4b6e53","Type":"ContainerStarted","Data":"986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f"} Jul 20 23:25:24.875709 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.875565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" event={"ID":"44cd07c0-db4a-4ca2-b045-01446e4b6e53","Type":"ContainerStarted","Data":"72c797b59cb72748802330cb8f7ca494abc52aaa9b4eb7390a7e0284ee609db3"} Jul 20 23:25:24.896036 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:24.895988 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" podStartSLOduration=0.895972535 podStartE2EDuration="895.972535ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:25:24.894752478 +0000 UTC m=+1900.213104776" watchObservedRunningTime="2026-07-20 23:25:24.895972535 +0000 UTC m=+1900.214324820" Jul 20 23:25:25.395644 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:25.395607 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:25.400417 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:25.400393 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:25.880043 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:25.880012 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:25.880983 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:25.880962 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:26.832878 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:26.832841 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk"] Jul 20 23:25:26.833471 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:26.833441 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="87153198-8a0f-439b-88a2-47dc53db5c0f" containerName="istio-proxy" Jul 20 23:25:26.836947 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:26.836923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:26.839851 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:26.839829 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2db333cd-serving-cert\"" Jul 20 23:25:26.848391 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:26.848366 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk"] Jul 20 23:25:26.940952 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:26.940907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b928-1a83-4734-a818-b41d9f6e2023-openshift-service-ca\") pod \"maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:26.941160 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:26.941007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3161b928-1a83-4734-a818-b41d9f6e2023-maas-api-tls\") pod \"maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:26.941160 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:26.941073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbb6\" (UniqueName: \"kubernetes.io/projected/3161b928-1a83-4734-a818-b41d9f6e2023-kube-api-access-ffbb6\") pod \"maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:27.042551 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.042497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b928-1a83-4734-a818-b41d9f6e2023-openshift-service-ca\") pod \"maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:27.042551 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.042565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3161b928-1a83-4734-a818-b41d9f6e2023-maas-api-tls\") pod \"maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:27.042834 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.042599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbb6\" (UniqueName: \"kubernetes.io/projected/3161b928-1a83-4734-a818-b41d9f6e2023-kube-api-access-ffbb6\") pod \"maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:27.043257 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.043230 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b928-1a83-4734-a818-b41d9f6e2023-openshift-service-ca\") pod \"maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:27.045123 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.045101 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3161b928-1a83-4734-a818-b41d9f6e2023-maas-api-tls\") pod \"maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:27.050946 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.050920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbb6\" (UniqueName: \"kubernetes.io/projected/3161b928-1a83-4734-a818-b41d9f6e2023-kube-api-access-ffbb6\") pod \"maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:27.148701 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.148661 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:27.285638 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.285614 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk"] Jul 20 23:25:27.288092 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:25:27.288059 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3161b928_1a83_4734_a818_b41d9f6e2023.slice/crio-b53d47b0efe6cd2f6b949bfd3f4790db3cb98785891a65800cd99ee8cd44d2a9 WatchSource:0}: Error finding container b53d47b0efe6cd2f6b949bfd3f4790db3cb98785891a65800cd99ee8cd44d2a9: Status 404 returned error can't find the container with id b53d47b0efe6cd2f6b949bfd3f4790db3cb98785891a65800cd99ee8cd44d2a9 Jul 20 23:25:27.289417 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.289398 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 23:25:27.890480 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.890369 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" event={"ID":"3161b928-1a83-4734-a818-b41d9f6e2023","Type":"ContainerStarted","Data":"77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4"} Jul 20 23:25:27.890866 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.890495 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" event={"ID":"3161b928-1a83-4734-a818-b41d9f6e2023","Type":"ContainerStarted","Data":"b53d47b0efe6cd2f6b949bfd3f4790db3cb98785891a65800cd99ee8cd44d2a9"} Jul 20 23:25:27.908250 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:27.908200 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" podStartSLOduration=1.574032165 podStartE2EDuration="1.908185644s" totalImagesPullingTime="334.153479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:25:27.906799745 +0000 UTC m=+1903.225152028" watchObservedRunningTime="2026-07-20 23:25:27.908185644 +0000 UTC m=+1903.226537930" Jul 20 23:25:28.895248 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:28.895219 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:32.714791 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.714748 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz"] Jul 20 23:25:32.718990 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.718965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.721903 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.721877 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2db333cd\"/\"openshift-service-ca.crt\"" Jul 20 23:25:32.722037 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.721921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2db333cd\"/\"kube-root-ca.crt\"" Jul 20 23:25:32.723978 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.723955 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2db333cd\"/\"default-dockercfg-xp6hv\"" Jul 20 23:25:32.724086 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.723966 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2db333cd\"/\"e2e-lifecycle-model-2db333cd-kserve-self-signed-certs\"" Jul 20 23:25:32.728340 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.728009 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz"] Jul 20 23:25:32.791905 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.791869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-dshm\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.792059 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.791921 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-model-cache\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.792059 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.791942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-home\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.792159 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.792052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-tmp-dir\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.792159 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.792092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0e80f7-5744-493e-aab5-32d8b27fc831-tls-certs\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.792159 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.792115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrdr\" (UniqueName: \"kubernetes.io/projected/5c0e80f7-5744-493e-aab5-32d8b27fc831-kube-api-access-lrrdr\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.892553 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.892517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-model-cache\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.892553 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.892557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-home\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.892808 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.892615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-tmp-dir\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.892808 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.892641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0e80f7-5744-493e-aab5-32d8b27fc831-tls-certs\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.892808 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.892659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrdr\" (UniqueName: \"kubernetes.io/projected/5c0e80f7-5744-493e-aab5-32d8b27fc831-kube-api-access-lrrdr\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.892808 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.892694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-dshm\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.893025 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.892986 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-model-cache\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.893083 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.893029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-home\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.893083 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.893073 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-tmp-dir\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.894966 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.894936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-dshm\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.895330 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.895303 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0e80f7-5744-493e-aab5-32d8b27fc831-tls-certs\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:32.901412 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:32.901387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrdr\" (UniqueName: \"kubernetes.io/projected/5c0e80f7-5744-493e-aab5-32d8b27fc831-kube-api-access-lrrdr\") pod \"e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:33.031110 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:33.031012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:33.212650 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:33.212620 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz"] Jul 20 23:25:33.215364 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:25:33.215337 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c0e80f7_5744_493e_aab5_32d8b27fc831.slice/crio-ccd87f44dff22e26201bd99594b6d2288a5730487903f0d8e4c943bc5b0315b6 WatchSource:0}: Error finding container ccd87f44dff22e26201bd99594b6d2288a5730487903f0d8e4c943bc5b0315b6: Status 404 returned error can't find the container with id ccd87f44dff22e26201bd99594b6d2288a5730487903f0d8e4c943bc5b0315b6 Jul 20 23:25:33.928361 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:33.928327 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" event={"ID":"5c0e80f7-5744-493e-aab5-32d8b27fc831","Type":"ContainerStarted","Data":"bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4"} Jul 20 23:25:33.928361 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:33.928367 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" event={"ID":"5c0e80f7-5744-493e-aab5-32d8b27fc831","Type":"ContainerStarted","Data":"ccd87f44dff22e26201bd99594b6d2288a5730487903f0d8e4c943bc5b0315b6"} Jul 20 23:25:33.946742 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:33.946688 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" podStartSLOduration=1.946671365 podStartE2EDuration="1.946671365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:25:33.944507492 +0000 UTC m=+1909.262859774" watchObservedRunningTime="2026-07-20 23:25:33.946671365 +0000 UTC m=+1909.265023651" Jul 20 23:25:34.906708 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:34.906680 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:25:39.959418 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:39.959384 2574 generic.go:356] "Generic (PLEG): container finished" podID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerID="986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f" exitCode=0 Jul 20 23:25:39.959935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:39.959470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" event={"ID":"44cd07c0-db4a-4ca2-b045-01446e4b6e53","Type":"ContainerDied","Data":"986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f"} Jul 20 23:25:39.959935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:39.959822 2574 scope.go:117] "RemoveContainer" containerID="986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f" Jul 20 23:25:39.960782 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:39.960750 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:25:39.960858 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:39.960838 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:25:39.960912 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:39.960890 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:25:40.880947 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:40.880889 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:40.969866 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:40.969153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" event={"ID":"44cd07c0-db4a-4ca2-b045-01446e4b6e53","Type":"ContainerStarted","Data":"1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb"} Jul 20 23:25:41.395661 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:41.395620 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:41.396896 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:41.396874 2574 patch_prober.go:28] interesting pod/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 20 23:25:41.396953 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:41.396921 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 20 23:25:42.396266 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:42.396227 2574 patch_prober.go:28] interesting pod/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 20 23:25:42.396686 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:42.396285 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 20 23:25:43.031531 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:43.031479 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:43.031753 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:43.031581 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:43.044318 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:43.044289 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:43.396418 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:43.396378 2574 patch_prober.go:28] interesting pod/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 20 23:25:43.396821 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:43.396441 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 20 23:25:44.000736 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:44.000710 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:25:44.396022 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:44.395986 2574 patch_prober.go:28] interesting pod/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 20 23:25:44.396223 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:44.396063 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 20 23:25:45.399567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:45.399537 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:45.400040 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:45.399945 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:25:45.400761 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:25:45.400741 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:26:14.755527 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:14.755488 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk"] Jul 20 23:26:14.756079 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:14.755707 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" podUID="3161b928-1a83-4734-a818-b41d9f6e2023" containerName="maas-api" containerID="cri-o://77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4" gracePeriod=30 Jul 20 23:26:14.901434 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:14.901396 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" podUID="3161b928-1a83-4734-a818-b41d9f6e2023" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.53:8443/health\": dial tcp 10.133.0.53:8443: connect: connection refused" Jul 20 23:26:15.018604 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.018539 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:26:15.095247 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.095209 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b928-1a83-4734-a818-b41d9f6e2023-openshift-service-ca\") pod \"3161b928-1a83-4734-a818-b41d9f6e2023\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " Jul 20 23:26:15.095247 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.095252 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffbb6\" (UniqueName: \"kubernetes.io/projected/3161b928-1a83-4734-a818-b41d9f6e2023-kube-api-access-ffbb6\") pod \"3161b928-1a83-4734-a818-b41d9f6e2023\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " Jul 20 23:26:15.095537 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.095323 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3161b928-1a83-4734-a818-b41d9f6e2023-maas-api-tls\") pod \"3161b928-1a83-4734-a818-b41d9f6e2023\" (UID: \"3161b928-1a83-4734-a818-b41d9f6e2023\") " Jul 20 23:26:15.095693 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.095669 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3161b928-1a83-4734-a818-b41d9f6e2023-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3161b928-1a83-4734-a818-b41d9f6e2023" (UID: "3161b928-1a83-4734-a818-b41d9f6e2023"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:26:15.097510 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.097486 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3161b928-1a83-4734-a818-b41d9f6e2023-kube-api-access-ffbb6" (OuterVolumeSpecName: "kube-api-access-ffbb6") pod "3161b928-1a83-4734-a818-b41d9f6e2023" (UID: "3161b928-1a83-4734-a818-b41d9f6e2023"). InnerVolumeSpecName "kube-api-access-ffbb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:26:15.097598 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.097493 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3161b928-1a83-4734-a818-b41d9f6e2023-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3161b928-1a83-4734-a818-b41d9f6e2023" (UID: "3161b928-1a83-4734-a818-b41d9f6e2023"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:26:15.142720 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.142684 2574 generic.go:356] "Generic (PLEG): container finished" podID="3161b928-1a83-4734-a818-b41d9f6e2023" containerID="77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4" exitCode=0 Jul 20 23:26:15.142893 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.142729 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" Jul 20 23:26:15.142893 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.142765 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" event={"ID":"3161b928-1a83-4734-a818-b41d9f6e2023","Type":"ContainerDied","Data":"77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4"} Jul 20 23:26:15.142893 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.142823 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk" event={"ID":"3161b928-1a83-4734-a818-b41d9f6e2023","Type":"ContainerDied","Data":"b53d47b0efe6cd2f6b949bfd3f4790db3cb98785891a65800cd99ee8cd44d2a9"} Jul 20 23:26:15.142893 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.142852 2574 scope.go:117] "RemoveContainer" containerID="77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4" Jul 20 23:26:15.157126 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.157105 2574 scope.go:117] "RemoveContainer" containerID="77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4" Jul 20 23:26:15.157435 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:26:15.157414 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4\": container with ID starting with 77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4 not found: ID does not exist" containerID="77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4" Jul 20 23:26:15.157527 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.157443 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4"} err="failed to get container status \"77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4\": rpc error: code = NotFound desc = could not find container \"77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4\": container with ID starting with 77274bb52ab8efba09d0c8dcc99495c245b3190cfb67be372eaf236ed4910bf4 not found: ID does not exist" Jul 20 23:26:15.166539 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.166514 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk"] Jul 20 23:26:15.170091 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.170066 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2db333cd-6987c468c4-bgvhk"] Jul 20 23:26:15.195920 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.195884 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3161b928-1a83-4734-a818-b41d9f6e2023-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:26:15.195920 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.195923 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3161b928-1a83-4734-a818-b41d9f6e2023-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:26:15.196076 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.195946 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffbb6\" (UniqueName: \"kubernetes.io/projected/3161b928-1a83-4734-a818-b41d9f6e2023-kube-api-access-ffbb6\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:26:15.227966 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:15.227938 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3161b928-1a83-4734-a818-b41d9f6e2023" path="/var/lib/kubelet/pods/3161b928-1a83-4734-a818-b41d9f6e2023/volumes" Jul 20 23:26:26.523706 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:26.523666 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz"] Jul 20 23:26:41.544272 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.544205 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" podUID="5c0e80f7-5744-493e-aab5-32d8b27fc831" containerName="main" containerID="cri-o://bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4" gracePeriod=45 Jul 20 23:26:41.802578 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.802512 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:26:41.942446 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942413 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0e80f7-5744-493e-aab5-32d8b27fc831-tls-certs\") pod \"5c0e80f7-5744-493e-aab5-32d8b27fc831\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " Jul 20 23:26:41.942446 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942446 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrrdr\" (UniqueName: \"kubernetes.io/projected/5c0e80f7-5744-493e-aab5-32d8b27fc831-kube-api-access-lrrdr\") pod \"5c0e80f7-5744-493e-aab5-32d8b27fc831\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " Jul 20 23:26:41.942708 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942490 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-model-cache\") pod \"5c0e80f7-5744-493e-aab5-32d8b27fc831\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " Jul 20 23:26:41.942708 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942537 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-home\") pod \"5c0e80f7-5744-493e-aab5-32d8b27fc831\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " Jul 20 23:26:41.942708 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942567 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-dshm\") pod \"5c0e80f7-5744-493e-aab5-32d8b27fc831\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " Jul 20 23:26:41.942708 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942606 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-tmp-dir\") pod \"5c0e80f7-5744-493e-aab5-32d8b27fc831\" (UID: \"5c0e80f7-5744-493e-aab5-32d8b27fc831\") " Jul 20 23:26:41.942913 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942829 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c0e80f7-5744-493e-aab5-32d8b27fc831" (UID: "5c0e80f7-5744-493e-aab5-32d8b27fc831"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:26:41.942913 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942864 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-home" (OuterVolumeSpecName: "home") pod "5c0e80f7-5744-493e-aab5-32d8b27fc831" (UID: "5c0e80f7-5744-493e-aab5-32d8b27fc831"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:26:41.942992 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942917 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c0e80f7-5744-493e-aab5-32d8b27fc831" (UID: "5c0e80f7-5744-493e-aab5-32d8b27fc831"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:26:41.942992 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.942936 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-model-cache\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:26:41.944835 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.944808 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c0e80f7-5744-493e-aab5-32d8b27fc831-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c0e80f7-5744-493e-aab5-32d8b27fc831" (UID: "5c0e80f7-5744-493e-aab5-32d8b27fc831"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:26:41.944957 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.944856 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0e80f7-5744-493e-aab5-32d8b27fc831-kube-api-access-lrrdr" (OuterVolumeSpecName: "kube-api-access-lrrdr") pod "5c0e80f7-5744-493e-aab5-32d8b27fc831" (UID: "5c0e80f7-5744-493e-aab5-32d8b27fc831"). InnerVolumeSpecName "kube-api-access-lrrdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:26:41.944957 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:41.944860 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-dshm" (OuterVolumeSpecName: "dshm") pod "5c0e80f7-5744-493e-aab5-32d8b27fc831" (UID: "5c0e80f7-5744-493e-aab5-32d8b27fc831"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:26:42.044407 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.044363 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c0e80f7-5744-493e-aab5-32d8b27fc831-tls-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:26:42.044407 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.044396 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrdr\" (UniqueName: \"kubernetes.io/projected/5c0e80f7-5744-493e-aab5-32d8b27fc831-kube-api-access-lrrdr\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:26:42.044407 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.044410 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-home\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:26:42.044407 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.044421 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-dshm\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:26:42.044718 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.044432 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c0e80f7-5744-493e-aab5-32d8b27fc831-tmp-dir\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:26:42.281260 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.281224 2574 generic.go:356] "Generic (PLEG): container finished" podID="5c0e80f7-5744-493e-aab5-32d8b27fc831" containerID="bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4" exitCode=0 Jul 20 23:26:42.281487 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.281270 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" Jul 20 23:26:42.281487 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.281300 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" event={"ID":"5c0e80f7-5744-493e-aab5-32d8b27fc831","Type":"ContainerDied","Data":"bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4"} Jul 20 23:26:42.281487 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.281341 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz" event={"ID":"5c0e80f7-5744-493e-aab5-32d8b27fc831","Type":"ContainerDied","Data":"ccd87f44dff22e26201bd99594b6d2288a5730487903f0d8e4c943bc5b0315b6"} Jul 20 23:26:42.281487 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.281365 2574 scope.go:117] "RemoveContainer" containerID="bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4" Jul 20 23:26:42.289560 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:26:42.289531 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c0e80f7_5744_493e_aab5_32d8b27fc831.slice/crio-ccd87f44dff22e26201bd99594b6d2288a5730487903f0d8e4c943bc5b0315b6\": RecentStats: unable to find data in memory cache]" Jul 20 23:26:42.290149 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.290130 2574 scope.go:117] "RemoveContainer" containerID="bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4" Jul 20 23:26:42.290448 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:26:42.290424 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4\": container with ID starting with bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4 not found: ID does not exist" containerID="bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4" Jul 20 23:26:42.290523 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.290482 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4"} err="failed to get container status \"bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4\": rpc error: code = NotFound desc = could not find container \"bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4\": container with ID starting with bb33f7f3f382c0f41ff64d4c2e824762db8b226ef15da2e905a582e72e1ac5f4 not found: ID does not exist" Jul 20 23:26:42.303358 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.303327 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz"] Jul 20 23:26:42.307119 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:42.307096 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2db333cd/e2e-lifecycle-model-2db333cd-kserve-54dc9c7b99-brthz"] Jul 20 23:26:43.227418 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:26:43.227382 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c0e80f7-5744-493e-aab5-32d8b27fc831" path="/var/lib/kubelet/pods/5c0e80f7-5744-493e-aab5-32d8b27fc831/volumes" Jul 20 23:27:15.142256 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:15.142216 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh"] Jul 20 23:27:15.142769 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:15.142505 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" containerID="cri-o://1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb" gracePeriod=30 Jul 20 23:27:15.403076 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:15.402991 2574 patch_prober.go:28] interesting pod/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 23:27:15.403076 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:15.403057 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 23:27:20.422328 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.422302 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:27:20.453035 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.452996 2574 generic.go:356] "Generic (PLEG): container finished" podID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerID="1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb" exitCode=0 Jul 20 23:27:20.453035 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.453050 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" Jul 20 23:27:20.453296 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.453082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" event={"ID":"44cd07c0-db4a-4ca2-b045-01446e4b6e53","Type":"ContainerDied","Data":"1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb"} Jul 20 23:27:20.453296 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.453136 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh" event={"ID":"44cd07c0-db4a-4ca2-b045-01446e4b6e53","Type":"ContainerDied","Data":"72c797b59cb72748802330cb8f7ca494abc52aaa9b4eb7390a7e0284ee609db3"} Jul 20 23:27:20.453296 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.453164 2574 scope.go:117] "RemoveContainer" containerID="1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb" Jul 20 23:27:20.461950 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.461930 2574 scope.go:117] "RemoveContainer" containerID="986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f" Jul 20 23:27:20.470678 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.470654 2574 scope.go:117] "RemoveContainer" containerID="1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb" Jul 20 23:27:20.470942 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:27:20.470925 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb\": container with ID starting with 1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb not found: ID does not exist" containerID="1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb" Jul 20 23:27:20.471007 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.470950 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb"} err="failed to get container status \"1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb\": rpc error: code = NotFound desc = could not find container \"1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb\": container with ID starting with 1e60f89a3f48ad686fe1de1380029916814dfa45332d7734d1b8d5ba56e513fb not found: ID does not exist" Jul 20 23:27:20.471007 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.470968 2574 scope.go:117] "RemoveContainer" containerID="986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f" Jul 20 23:27:20.471207 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:27:20.471186 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f\": container with ID starting with 986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f not found: ID does not exist" containerID="986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f" Jul 20 23:27:20.471281 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.471218 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f"} err="failed to get container status \"986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f\": rpc error: code = NotFound desc = could not find container \"986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f\": container with ID starting with 986984ef61a33f1593e7ae3429854aed942dde9742753e102200b879bd7ee81f not found: ID does not exist" Jul 20 23:27:20.481533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481506 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-socket\") pod \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " Jul 20 23:27:20.481786 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481555 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-token\") pod \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " Jul 20 23:27:20.481786 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481599 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istiod-ca-cert\") pod \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " Jul 20 23:27:20.481786 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481653 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-podinfo\") pod \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " Jul 20 23:27:20.481786 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481680 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8qhd\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-kube-api-access-b8qhd\") pod \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " Jul 20 23:27:20.481786 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481707 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-envoy\") pod \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " Jul 20 23:27:20.481786 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481751 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-certs\") pod \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " Jul 20 23:27:20.481786 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481782 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-credential-socket\") pod \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " Jul 20 23:27:20.482369 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481804 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-data\") pod \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\" (UID: \"44cd07c0-db4a-4ca2-b045-01446e4b6e53\") " Jul 20 23:27:20.482369 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.481854 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44cd07c0-db4a-4ca2-b045-01446e4b6e53" (UID: "44cd07c0-db4a-4ca2-b045-01446e4b6e53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:27:20.482369 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.482079 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:27:20.482369 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.482131 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44cd07c0-db4a-4ca2-b045-01446e4b6e53" (UID: "44cd07c0-db4a-4ca2-b045-01446e4b6e53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:27:20.482369 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.482246 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44cd07c0-db4a-4ca2-b045-01446e4b6e53" (UID: "44cd07c0-db4a-4ca2-b045-01446e4b6e53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:27:20.482369 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.482308 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44cd07c0-db4a-4ca2-b045-01446e4b6e53" (UID: "44cd07c0-db4a-4ca2-b045-01446e4b6e53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:27:20.482977 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.482954 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-data" (OuterVolumeSpecName: "istio-data") pod "44cd07c0-db4a-4ca2-b045-01446e4b6e53" (UID: "44cd07c0-db4a-4ca2-b045-01446e4b6e53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:27:20.484176 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.484148 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44cd07c0-db4a-4ca2-b045-01446e4b6e53" (UID: "44cd07c0-db4a-4ca2-b045-01446e4b6e53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:27:20.484348 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.484314 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44cd07c0-db4a-4ca2-b045-01446e4b6e53" (UID: "44cd07c0-db4a-4ca2-b045-01446e4b6e53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 23:27:20.484440 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.484354 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-token" (OuterVolumeSpecName: "istio-token") pod "44cd07c0-db4a-4ca2-b045-01446e4b6e53" (UID: "44cd07c0-db4a-4ca2-b045-01446e4b6e53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:27:20.484440 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.484389 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-kube-api-access-b8qhd" (OuterVolumeSpecName: "kube-api-access-b8qhd") pod "44cd07c0-db4a-4ca2-b045-01446e4b6e53" (UID: "44cd07c0-db4a-4ca2-b045-01446e4b6e53"). InnerVolumeSpecName "kube-api-access-b8qhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:27:20.582825 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.582785 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-podinfo\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:27:20.582825 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.582819 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8qhd\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-kube-api-access-b8qhd\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:27:20.582825 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.582833 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-envoy\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:27:20.583072 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.582845 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-workload-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:27:20.583072 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.582856 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-credential-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:27:20.583072 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.582867 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-data\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:27:20.583072 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.582880 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istio-token\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:27:20.583072 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.582890 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44cd07c0-db4a-4ca2-b045-01446e4b6e53-istiod-ca-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:27:20.778225 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.778191 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh"] Jul 20 23:27:20.782713 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:20.782678 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2db333cd-openshift-default-7848db79cf-l47xh"] Jul 20 23:27:21.229291 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:27:21.229248 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" path="/var/lib/kubelet/pods/44cd07c0-db4a-4ca2-b045-01446e4b6e53/volumes" Jul 20 23:28:10.961087 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.960994 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg"] Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961359 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961370 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961381 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c0e80f7-5744-493e-aab5-32d8b27fc831" containerName="main" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961387 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c0e80f7-5744-493e-aab5-32d8b27fc831" containerName="main" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961396 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961401 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961417 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3161b928-1a83-4734-a818-b41d9f6e2023" containerName="maas-api" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961422 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3161b928-1a83-4734-a818-b41d9f6e2023" containerName="maas-api" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961485 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3161b928-1a83-4734-a818-b41d9f6e2023" containerName="maas-api" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961497 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961503 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c0e80f7-5744-493e-aab5-32d8b27fc831" containerName="main" Jul 20 23:28:10.961545 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.961509 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="44cd07c0-db4a-4ca2-b045-01446e4b6e53" containerName="istio-proxy" Jul 20 23:28:10.964403 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.964387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:10.967200 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.967176 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6a0c93-openshift-default-dockercfg-jwwc6\"" Jul 20 23:28:10.978649 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:10.978621 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg"] Jul 20 23:28:11.034966 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.034920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-data\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.034966 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.034960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-istio-token\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.035180 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.035009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-credential-socket\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.035180 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.035055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-socket\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.035180 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.035076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a772-e13a-4813-8628-5bb317027aeb-istio-podinfo\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.035180 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.035092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-envoy\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.035180 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.035120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrjjn\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-kube-api-access-vrjjn\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.035180 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.035136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-certs\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.035385 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.035181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a772-e13a-4813-8628-5bb317027aeb-istiod-ca-cert\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136237 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrjjn\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-kube-api-access-vrjjn\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136237 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-certs\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a772-e13a-4813-8628-5bb317027aeb-istiod-ca-cert\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-data\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-istio-token\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-credential-socket\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136387 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-socket\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a772-e13a-4813-8628-5bb317027aeb-istio-podinfo\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-envoy\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136946 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136711 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-certs\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.136946 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136750 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-data\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.137069 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.136945 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-socket\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.137069 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.137015 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-credential-socket\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.137216 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.137194 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a772-e13a-4813-8628-5bb317027aeb-istiod-ca-cert\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.138749 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.138721 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-envoy\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.139023 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.139004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a772-e13a-4813-8628-5bb317027aeb-istio-podinfo\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.144771 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.144743 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-istio-token\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.145206 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.145188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrjjn\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-kube-api-access-vrjjn\") pod \"e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.274448 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.274360 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:11.417578 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.417551 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg"] Jul 20 23:28:11.419223 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:28:11.419192 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f99a772_e13a_4813_8628_5bb317027aeb.slice/crio-61869c87c9ca4b020ba863c67e94417b3920a8298bd20918f755354286ef2c47 WatchSource:0}: Error finding container 61869c87c9ca4b020ba863c67e94417b3920a8298bd20918f755354286ef2c47: Status 404 returned error can't find the container with id 61869c87c9ca4b020ba863c67e94417b3920a8298bd20918f755354286ef2c47 Jul 20 23:28:11.421312 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.421279 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:28:11.421383 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.421345 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:28:11.421383 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.421375 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:28:11.693011 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.692976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" event={"ID":"6f99a772-e13a-4813-8628-5bb317027aeb","Type":"ContainerStarted","Data":"3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247"} Jul 20 23:28:11.693199 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.693033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" event={"ID":"6f99a772-e13a-4813-8628-5bb317027aeb","Type":"ContainerStarted","Data":"61869c87c9ca4b020ba863c67e94417b3920a8298bd20918f755354286ef2c47"} Jul 20 23:28:11.713401 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:11.713349 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" podStartSLOduration=1.7133336 podStartE2EDuration="1.7133336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:28:11.712261196 +0000 UTC m=+2067.030613482" watchObservedRunningTime="2026-07-20 23:28:11.7133336 +0000 UTC m=+2067.031685886" Jul 20 23:28:12.274721 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:12.274688 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:12.276111 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:12.276085 2574 patch_prober.go:28] interesting pod/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 20 23:28:12.276245 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:12.276146 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 20 23:28:13.279181 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.279143 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:13.703131 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.703100 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:13.704107 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.704081 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:13.704553 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.704530 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g"] Jul 20 23:28:13.708723 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.708703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:13.711342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.711308 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6a0c93-serving-cert\"" Jul 20 23:28:13.721920 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.721894 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g"] Jul 20 23:28:13.761490 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.761421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124c0efa-d056-4891-b18b-aa6762c741d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:13.761490 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.761474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124c0efa-d056-4891-b18b-aa6762c741d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:13.761743 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.761623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhrc\" (UniqueName: \"kubernetes.io/projected/124c0efa-d056-4891-b18b-aa6762c741d8-kube-api-access-ldhrc\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:13.862259 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.862222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124c0efa-d056-4891-b18b-aa6762c741d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:13.862259 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.862263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124c0efa-d056-4891-b18b-aa6762c741d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:13.862537 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.862343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhrc\" (UniqueName: \"kubernetes.io/projected/124c0efa-d056-4891-b18b-aa6762c741d8-kube-api-access-ldhrc\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:13.862537 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:28:13.862377 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-serving-cert: secret "maas-api-e2e-inf-a-6a0c93-serving-cert" not found Jul 20 23:28:13.862537 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:28:13.862436 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/124c0efa-d056-4891-b18b-aa6762c741d8-maas-api-tls podName:124c0efa-d056-4891-b18b-aa6762c741d8 nodeName:}" failed. No retries permitted until 2026-07-20 23:28:14.3624202 +0000 UTC m=+2069.680772464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/124c0efa-d056-4891-b18b-aa6762c741d8-maas-api-tls") pod "maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" (UID: "124c0efa-d056-4891-b18b-aa6762c741d8") : secret "maas-api-e2e-inf-a-6a0c93-serving-cert" not found Jul 20 23:28:13.862957 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.862938 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124c0efa-d056-4891-b18b-aa6762c741d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:13.871239 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:13.871208 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhrc\" (UniqueName: \"kubernetes.io/projected/124c0efa-d056-4891-b18b-aa6762c741d8-kube-api-access-ldhrc\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:14.366819 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:14.366788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124c0efa-d056-4891-b18b-aa6762c741d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:14.369263 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:14.369241 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124c0efa-d056-4891-b18b-aa6762c741d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:14.620052 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:14.619956 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:14.767129 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:14.767096 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g"] Jul 20 23:28:14.768781 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:28:14.768747 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124c0efa_d056_4891_b18b_aa6762c741d8.slice/crio-4b9c35945e521e2e2b58411ee9d11d02783129d5b5e941cbc059fce1a51dc30c WatchSource:0}: Error finding container 4b9c35945e521e2e2b58411ee9d11d02783129d5b5e941cbc059fce1a51dc30c: Status 404 returned error can't find the container with id 4b9c35945e521e2e2b58411ee9d11d02783129d5b5e941cbc059fce1a51dc30c Jul 20 23:28:15.713821 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:15.713777 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" event={"ID":"124c0efa-d056-4891-b18b-aa6762c741d8","Type":"ContainerStarted","Data":"750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c"} Jul 20 23:28:15.714241 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:15.713843 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" event={"ID":"124c0efa-d056-4891-b18b-aa6762c741d8","Type":"ContainerStarted","Data":"4b9c35945e521e2e2b58411ee9d11d02783129d5b5e941cbc059fce1a51dc30c"} Jul 20 23:28:15.714241 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:15.713977 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:15.732162 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:15.732102 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" podStartSLOduration=2.286182286 podStartE2EDuration="2.732087706s" totalImagesPullingTime="445.90542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:28:15.730993797 +0000 UTC m=+2071.049346083" watchObservedRunningTime="2026-07-20 23:28:15.732087706 +0000 UTC m=+2071.050439993" Jul 20 23:28:21.538329 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.538292 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j"] Jul 20 23:28:21.543026 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.542996 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:21.545884 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.545863 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-324c54-serving-cert\"" Jul 20 23:28:21.552138 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.552111 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j"] Jul 20 23:28:21.637295 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.637247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjwss\" (UniqueName: \"kubernetes.io/projected/88a2e127-740e-4f53-a82d-a3217a0412b6-kube-api-access-qjwss\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:21.637543 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.637333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a2e127-740e-4f53-a82d-a3217a0412b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:21.637543 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.637396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a2e127-740e-4f53-a82d-a3217a0412b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:21.725795 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.725759 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:28:21.738780 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.738745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a2e127-740e-4f53-a82d-a3217a0412b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:21.738780 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.738781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a2e127-740e-4f53-a82d-a3217a0412b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:21.739073 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.738848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjwss\" (UniqueName: \"kubernetes.io/projected/88a2e127-740e-4f53-a82d-a3217a0412b6-kube-api-access-qjwss\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:21.739073 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:28:21.738885 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-serving-cert: secret "maas-api-e2e-inf-b-324c54-serving-cert" not found Jul 20 23:28:21.739073 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:28:21.738957 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a2e127-740e-4f53-a82d-a3217a0412b6-maas-api-tls podName:88a2e127-740e-4f53-a82d-a3217a0412b6 nodeName:}" failed. No retries permitted until 2026-07-20 23:28:22.238940055 +0000 UTC m=+2077.557292325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88a2e127-740e-4f53-a82d-a3217a0412b6-maas-api-tls") pod "maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" (UID: "88a2e127-740e-4f53-a82d-a3217a0412b6") : secret "maas-api-e2e-inf-b-324c54-serving-cert" not found Jul 20 23:28:21.739582 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.739548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a2e127-740e-4f53-a82d-a3217a0412b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:21.754507 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:21.754451 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjwss\" (UniqueName: \"kubernetes.io/projected/88a2e127-740e-4f53-a82d-a3217a0412b6-kube-api-access-qjwss\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:22.244651 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:22.244604 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a2e127-740e-4f53-a82d-a3217a0412b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:22.247039 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:22.247013 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a2e127-740e-4f53-a82d-a3217a0412b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:22.456657 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:22.456616 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:22.646553 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:22.646529 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j"] Jul 20 23:28:22.648454 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:28:22.648425 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a2e127_740e_4f53_a82d_a3217a0412b6.slice/crio-37f23b19daa5a58a3e65131a08cd50ed35c029c813b572f2837df7fead6eefcb WatchSource:0}: Error finding container 37f23b19daa5a58a3e65131a08cd50ed35c029c813b572f2837df7fead6eefcb: Status 404 returned error can't find the container with id 37f23b19daa5a58a3e65131a08cd50ed35c029c813b572f2837df7fead6eefcb Jul 20 23:28:22.747905 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:22.747863 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" event={"ID":"88a2e127-740e-4f53-a82d-a3217a0412b6","Type":"ContainerStarted","Data":"37f23b19daa5a58a3e65131a08cd50ed35c029c813b572f2837df7fead6eefcb"} Jul 20 23:28:23.754607 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:23.754573 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" event={"ID":"88a2e127-740e-4f53-a82d-a3217a0412b6","Type":"ContainerStarted","Data":"def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae"} Jul 20 23:28:23.755119 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:23.754677 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:23.773342 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:23.773268 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" podStartSLOduration=2.438740218 podStartE2EDuration="2.77324394s" totalImagesPullingTime="334.503722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:28:23.771629589 +0000 UTC m=+2079.089981877" watchObservedRunningTime="2026-07-20 23:28:23.77324394 +0000 UTC m=+2079.091596230" Jul 20 23:28:27.775567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:27.775481 2574 generic.go:356] "Generic (PLEG): container finished" podID="6f99a772-e13a-4813-8628-5bb317027aeb" containerID="3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247" exitCode=0 Jul 20 23:28:27.775567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:27.775531 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" event={"ID":"6f99a772-e13a-4813-8628-5bb317027aeb","Type":"ContainerDied","Data":"3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247"} Jul 20 23:28:27.776013 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:27.775929 2574 scope.go:117] "RemoveContainer" containerID="3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247" Jul 20 23:28:27.776903 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:27.776870 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:28:27.777000 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:27.776939 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:28:27.777000 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:27.776972 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:28:28.703168 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:28.703128 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:28.785651 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:28.785604 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" event={"ID":"6f99a772-e13a-4813-8628-5bb317027aeb","Type":"ContainerStarted","Data":"27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2"} Jul 20 23:28:29.275377 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:29.275338 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:29.276825 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:29.276790 2574 patch_prober.go:28] interesting pod/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 20 23:28:29.276964 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:29.276860 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 20 23:28:29.773805 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:29.773769 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:28:30.275485 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:30.275426 2574 patch_prober.go:28] interesting pod/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 20 23:28:30.275998 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:30.275516 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 20 23:28:31.275129 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:31.275090 2574 patch_prober.go:28] interesting pod/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 20 23:28:31.275446 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:31.275414 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 20 23:28:32.275590 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:32.275542 2574 patch_prober.go:28] interesting pod/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 20 23:28:32.276091 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:32.275632 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 20 23:28:33.279524 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:33.279488 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:33.279890 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:33.279760 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:33.280331 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:33.280314 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:28:45.206228 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:45.206194 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:28:45.210630 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:45.210608 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:28:50.815677 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.815644 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq"] Jul 20 23:28:50.819179 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.819159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:50.823235 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.823216 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6a0c93\"/\"default-dockercfg-fjbkq\"" Jul 20 23:28:50.823381 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.823292 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6a0c93\"/\"test-model-6a0c93-kserve-self-signed-certs\"" Jul 20 23:28:50.823474 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.823372 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6a0c93\"/\"openshift-service-ca.crt\"" Jul 20 23:28:50.823474 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.823422 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6a0c93\"/\"kube-root-ca.crt\"" Jul 20 23:28:50.829328 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.829301 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq"] Jul 20 23:28:50.906936 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.906890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-tmp-dir\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:50.906936 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.906944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-home\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:50.907211 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.906959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-dshm\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:50.907211 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.907033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7ll\" (UniqueName: \"kubernetes.io/projected/db302c56-2c83-4b55-aa95-200c16138598-kube-api-access-vw7ll\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:50.907211 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.907078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-model-cache\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:50.907211 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:50.907111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db302c56-2c83-4b55-aa95-200c16138598-tls-certs\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.008525 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.008485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-tmp-dir\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.008725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.008559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-home\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.008725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.008579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-dshm\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.008725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.008614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7ll\" (UniqueName: \"kubernetes.io/projected/db302c56-2c83-4b55-aa95-200c16138598-kube-api-access-vw7ll\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.008725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.008638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-model-cache\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.008725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.008669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db302c56-2c83-4b55-aa95-200c16138598-tls-certs\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.009012 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.008898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-tmp-dir\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.009070 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.009000 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-home\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.009070 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.009028 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-model-cache\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.010986 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.010965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-dshm\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.011145 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.011127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db302c56-2c83-4b55-aa95-200c16138598-tls-certs\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.016631 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.016609 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7ll\" (UniqueName: \"kubernetes.io/projected/db302c56-2c83-4b55-aa95-200c16138598-kube-api-access-vw7ll\") pod \"test-model-6a0c93-kserve-578958cc57-dzmfq\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.128860 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.128821 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:28:51.265593 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.265564 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq"] Jul 20 23:28:51.268176 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:28:51.268142 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb302c56_2c83_4b55_aa95_200c16138598.slice/crio-423d71a4ed717bd5e698a5b2d98b35127af9c3bb85b5a94cf509a521a28f1468 WatchSource:0}: Error finding container 423d71a4ed717bd5e698a5b2d98b35127af9c3bb85b5a94cf509a521a28f1468: Status 404 returned error can't find the container with id 423d71a4ed717bd5e698a5b2d98b35127af9c3bb85b5a94cf509a521a28f1468 Jul 20 23:28:51.900305 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.900261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" event={"ID":"db302c56-2c83-4b55-aa95-200c16138598","Type":"ContainerStarted","Data":"dd6a6034ac621eea881f877758f4029eb931e20f955dbc033f8d7a7648ccf795"} Jul 20 23:28:51.900710 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.900314 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" event={"ID":"db302c56-2c83-4b55-aa95-200c16138598","Type":"ContainerStarted","Data":"423d71a4ed717bd5e698a5b2d98b35127af9c3bb85b5a94cf509a521a28f1468"} Jul 20 23:28:51.918154 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:28:51.918095 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" podStartSLOduration=1.918078987 podStartE2EDuration="1.918078987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:28:51.916336003 +0000 UTC m=+2107.234688352" watchObservedRunningTime="2026-07-20 23:28:51.918078987 +0000 UTC m=+2107.236431274" Jul 20 23:29:01.129981 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:29:01.129937 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:29:01.130511 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:29:01.130007 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:29:01.143032 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:29:01.143010 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:29:01.962055 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:29:01.962025 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:30:06.098725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:06.098687 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq"] Jul 20 23:30:17.530446 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:17.530411 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g"] Jul 20 23:30:17.531050 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:17.530630 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" podUID="124c0efa-d056-4891-b18b-aa6762c741d8" containerName="maas-api" containerID="cri-o://750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c" gracePeriod=30 Jul 20 23:30:17.977870 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:17.977845 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:30:18.015413 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.015381 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124c0efa-d056-4891-b18b-aa6762c741d8-maas-api-tls\") pod \"124c0efa-d056-4891-b18b-aa6762c741d8\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " Jul 20 23:30:18.015413 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.015419 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124c0efa-d056-4891-b18b-aa6762c741d8-openshift-service-ca\") pod \"124c0efa-d056-4891-b18b-aa6762c741d8\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " Jul 20 23:30:18.015770 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.015439 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldhrc\" (UniqueName: \"kubernetes.io/projected/124c0efa-d056-4891-b18b-aa6762c741d8-kube-api-access-ldhrc\") pod \"124c0efa-d056-4891-b18b-aa6762c741d8\" (UID: \"124c0efa-d056-4891-b18b-aa6762c741d8\") " Jul 20 23:30:18.015863 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.015837 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124c0efa-d056-4891-b18b-aa6762c741d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "124c0efa-d056-4891-b18b-aa6762c741d8" (UID: "124c0efa-d056-4891-b18b-aa6762c741d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:30:18.017603 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.017572 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124c0efa-d056-4891-b18b-aa6762c741d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "124c0efa-d056-4891-b18b-aa6762c741d8" (UID: "124c0efa-d056-4891-b18b-aa6762c741d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:30:18.017709 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.017654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124c0efa-d056-4891-b18b-aa6762c741d8-kube-api-access-ldhrc" (OuterVolumeSpecName: "kube-api-access-ldhrc") pod "124c0efa-d056-4891-b18b-aa6762c741d8" (UID: "124c0efa-d056-4891-b18b-aa6762c741d8"). InnerVolumeSpecName "kube-api-access-ldhrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:30:18.116841 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.116803 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124c0efa-d056-4891-b18b-aa6762c741d8-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:18.116841 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.116837 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124c0efa-d056-4891-b18b-aa6762c741d8-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:18.116841 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.116847 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhrc\" (UniqueName: \"kubernetes.io/projected/124c0efa-d056-4891-b18b-aa6762c741d8-kube-api-access-ldhrc\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:18.318891 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.318853 2574 generic.go:356] "Generic (PLEG): container finished" podID="124c0efa-d056-4891-b18b-aa6762c741d8" containerID="750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c" exitCode=0 Jul 20 23:30:18.318891 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.318881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" event={"ID":"124c0efa-d056-4891-b18b-aa6762c741d8","Type":"ContainerDied","Data":"750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c"} Jul 20 23:30:18.319126 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.318892 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" Jul 20 23:30:18.319126 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.318934 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g" event={"ID":"124c0efa-d056-4891-b18b-aa6762c741d8","Type":"ContainerDied","Data":"4b9c35945e521e2e2b58411ee9d11d02783129d5b5e941cbc059fce1a51dc30c"} Jul 20 23:30:18.319126 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.318969 2574 scope.go:117] "RemoveContainer" containerID="750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c" Jul 20 23:30:18.327767 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.327748 2574 scope.go:117] "RemoveContainer" containerID="750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c" Jul 20 23:30:18.328033 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:30:18.328007 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c\": container with ID starting with 750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c not found: ID does not exist" containerID="750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c" Jul 20 23:30:18.328104 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.328047 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c"} err="failed to get container status \"750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c\": rpc error: code = NotFound desc = could not find container \"750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c\": container with ID starting with 750eaf9ea41c19f7067fd721417bd07bb9572a19405e088ca1972b122ac3183c not found: ID does not exist" Jul 20 23:30:18.340778 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.340747 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g"] Jul 20 23:30:18.345728 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:18.345700 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6a0c93-7fc866f678-wzs6g"] Jul 20 23:30:19.228772 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:19.228739 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="124c0efa-d056-4891-b18b-aa6762c741d8" path="/var/lib/kubelet/pods/124c0efa-d056-4891-b18b-aa6762c741d8/volumes" Jul 20 23:30:21.120646 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.120583 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" podUID="db302c56-2c83-4b55-aa95-200c16138598" containerName="main" containerID="cri-o://dd6a6034ac621eea881f877758f4029eb931e20f955dbc033f8d7a7648ccf795" gracePeriod=45 Jul 20 23:30:21.336349 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.336308 2574 generic.go:356] "Generic (PLEG): container finished" podID="db302c56-2c83-4b55-aa95-200c16138598" containerID="dd6a6034ac621eea881f877758f4029eb931e20f955dbc033f8d7a7648ccf795" exitCode=0 Jul 20 23:30:21.336536 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.336375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" event={"ID":"db302c56-2c83-4b55-aa95-200c16138598","Type":"ContainerDied","Data":"dd6a6034ac621eea881f877758f4029eb931e20f955dbc033f8d7a7648ccf795"} Jul 20 23:30:21.373473 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.373397 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:30:21.446639 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.446609 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-model-cache\") pod \"db302c56-2c83-4b55-aa95-200c16138598\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " Jul 20 23:30:21.446821 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.446678 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-dshm\") pod \"db302c56-2c83-4b55-aa95-200c16138598\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " Jul 20 23:30:21.446821 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.446695 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-home\") pod \"db302c56-2c83-4b55-aa95-200c16138598\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " Jul 20 23:30:21.446821 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.446711 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db302c56-2c83-4b55-aa95-200c16138598-tls-certs\") pod \"db302c56-2c83-4b55-aa95-200c16138598\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " Jul 20 23:30:21.446821 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.446743 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-tmp-dir\") pod \"db302c56-2c83-4b55-aa95-200c16138598\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " Jul 20 23:30:21.446821 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.446758 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw7ll\" (UniqueName: \"kubernetes.io/projected/db302c56-2c83-4b55-aa95-200c16138598-kube-api-access-vw7ll\") pod \"db302c56-2c83-4b55-aa95-200c16138598\" (UID: \"db302c56-2c83-4b55-aa95-200c16138598\") " Jul 20 23:30:21.447096 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.446852 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-model-cache" (OuterVolumeSpecName: "model-cache") pod "db302c56-2c83-4b55-aa95-200c16138598" (UID: "db302c56-2c83-4b55-aa95-200c16138598"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:21.447096 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.446991 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-home" (OuterVolumeSpecName: "home") pod "db302c56-2c83-4b55-aa95-200c16138598" (UID: "db302c56-2c83-4b55-aa95-200c16138598"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:21.447096 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.447030 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-model-cache\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:21.447096 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.447057 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db302c56-2c83-4b55-aa95-200c16138598" (UID: "db302c56-2c83-4b55-aa95-200c16138598"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:21.448947 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.448915 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db302c56-2c83-4b55-aa95-200c16138598-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db302c56-2c83-4b55-aa95-200c16138598" (UID: "db302c56-2c83-4b55-aa95-200c16138598"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:30:21.449054 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.448966 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-dshm" (OuterVolumeSpecName: "dshm") pod "db302c56-2c83-4b55-aa95-200c16138598" (UID: "db302c56-2c83-4b55-aa95-200c16138598"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:21.449054 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.448989 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db302c56-2c83-4b55-aa95-200c16138598-kube-api-access-vw7ll" (OuterVolumeSpecName: "kube-api-access-vw7ll") pod "db302c56-2c83-4b55-aa95-200c16138598" (UID: "db302c56-2c83-4b55-aa95-200c16138598"). InnerVolumeSpecName "kube-api-access-vw7ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:30:21.547517 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.547450 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-tmp-dir\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:21.547517 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.547511 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7ll\" (UniqueName: \"kubernetes.io/projected/db302c56-2c83-4b55-aa95-200c16138598-kube-api-access-vw7ll\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:21.547517 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.547523 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-dshm\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:21.547753 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.547537 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db302c56-2c83-4b55-aa95-200c16138598-home\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:21.547753 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:21.547547 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db302c56-2c83-4b55-aa95-200c16138598-tls-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:22.342045 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:22.342007 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" event={"ID":"db302c56-2c83-4b55-aa95-200c16138598","Type":"ContainerDied","Data":"423d71a4ed717bd5e698a5b2d98b35127af9c3bb85b5a94cf509a521a28f1468"} Jul 20 23:30:22.342045 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:22.342032 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq" Jul 20 23:30:22.342722 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:22.342058 2574 scope.go:117] "RemoveContainer" containerID="dd6a6034ac621eea881f877758f4029eb931e20f955dbc033f8d7a7648ccf795" Jul 20 23:30:22.363240 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:22.363211 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq"] Jul 20 23:30:22.367247 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:22.367226 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6a0c93/test-model-6a0c93-kserve-578958cc57-dzmfq"] Jul 20 23:30:23.228248 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:23.228218 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db302c56-2c83-4b55-aa95-200c16138598" path="/var/lib/kubelet/pods/db302c56-2c83-4b55-aa95-200c16138598/volumes" Jul 20 23:30:28.717567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:28.717532 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg"] Jul 20 23:30:28.718026 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:28.717742 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" containerID="cri-o://27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2" gracePeriod=30 Jul 20 23:30:33.282439 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:33.282397 2574 patch_prober.go:28] interesting pod/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 23:30:33.282851 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:33.282507 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 23:30:33.965604 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:33.965580 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:30:34.057381 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057298 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-data\") pod \"6f99a772-e13a-4813-8628-5bb317027aeb\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " Jul 20 23:30:34.057381 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057364 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-certs\") pod \"6f99a772-e13a-4813-8628-5bb317027aeb\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " Jul 20 23:30:34.057636 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057395 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a772-e13a-4813-8628-5bb317027aeb-istio-podinfo\") pod \"6f99a772-e13a-4813-8628-5bb317027aeb\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " Jul 20 23:30:34.057636 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057423 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-istio-token\") pod \"6f99a772-e13a-4813-8628-5bb317027aeb\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " Jul 20 23:30:34.057636 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057443 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-socket\") pod \"6f99a772-e13a-4813-8628-5bb317027aeb\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " Jul 20 23:30:34.057636 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057471 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-envoy\") pod \"6f99a772-e13a-4813-8628-5bb317027aeb\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " Jul 20 23:30:34.057636 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057490 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a772-e13a-4813-8628-5bb317027aeb-istiod-ca-cert\") pod \"6f99a772-e13a-4813-8628-5bb317027aeb\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " Jul 20 23:30:34.057636 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057565 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrjjn\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-kube-api-access-vrjjn\") pod \"6f99a772-e13a-4813-8628-5bb317027aeb\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " Jul 20 23:30:34.057636 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057616 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-credential-socket\") pod \"6f99a772-e13a-4813-8628-5bb317027aeb\" (UID: \"6f99a772-e13a-4813-8628-5bb317027aeb\") " Jul 20 23:30:34.058007 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057648 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f99a772-e13a-4813-8628-5bb317027aeb" (UID: "6f99a772-e13a-4813-8628-5bb317027aeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:34.058007 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057829 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f99a772-e13a-4813-8628-5bb317027aeb" (UID: "6f99a772-e13a-4813-8628-5bb317027aeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:34.058007 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057879 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:34.058007 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.057944 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f99a772-e13a-4813-8628-5bb317027aeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f99a772-e13a-4813-8628-5bb317027aeb" (UID: "6f99a772-e13a-4813-8628-5bb317027aeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:30:34.058224 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.058027 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f99a772-e13a-4813-8628-5bb317027aeb" (UID: "6f99a772-e13a-4813-8628-5bb317027aeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:34.058224 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.058180 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f99a772-e13a-4813-8628-5bb317027aeb" (UID: "6f99a772-e13a-4813-8628-5bb317027aeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:34.059849 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.059788 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f99a772-e13a-4813-8628-5bb317027aeb" (UID: "6f99a772-e13a-4813-8628-5bb317027aeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:34.059849 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.059815 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f99a772-e13a-4813-8628-5bb317027aeb" (UID: "6f99a772-e13a-4813-8628-5bb317027aeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:30:34.059849 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.059831 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-kube-api-access-vrjjn" (OuterVolumeSpecName: "kube-api-access-vrjjn") pod "6f99a772-e13a-4813-8628-5bb317027aeb" (UID: "6f99a772-e13a-4813-8628-5bb317027aeb"). InnerVolumeSpecName "kube-api-access-vrjjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:30:34.060025 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.059864 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f99a772-e13a-4813-8628-5bb317027aeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f99a772-e13a-4813-8628-5bb317027aeb" (UID: "6f99a772-e13a-4813-8628-5bb317027aeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 23:30:34.158597 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.158561 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-istio-token\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:34.158597 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.158590 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-workload-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:34.158597 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.158599 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-envoy\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:34.158597 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.158607 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a772-e13a-4813-8628-5bb317027aeb-istiod-ca-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:34.158876 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.158616 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrjjn\" (UniqueName: \"kubernetes.io/projected/6f99a772-e13a-4813-8628-5bb317027aeb-kube-api-access-vrjjn\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:34.158876 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.158627 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-credential-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:34.158876 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.158639 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a772-e13a-4813-8628-5bb317027aeb-istio-data\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:34.158876 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.158650 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a772-e13a-4813-8628-5bb317027aeb-istio-podinfo\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:34.400956 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.400923 2574 generic.go:356] "Generic (PLEG): container finished" podID="6f99a772-e13a-4813-8628-5bb317027aeb" containerID="27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2" exitCode=0 Jul 20 23:30:34.401351 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.400970 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" Jul 20 23:30:34.401351 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.401015 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" event={"ID":"6f99a772-e13a-4813-8628-5bb317027aeb","Type":"ContainerDied","Data":"27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2"} Jul 20 23:30:34.401351 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.401073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg" event={"ID":"6f99a772-e13a-4813-8628-5bb317027aeb","Type":"ContainerDied","Data":"61869c87c9ca4b020ba863c67e94417b3920a8298bd20918f755354286ef2c47"} Jul 20 23:30:34.401351 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.401103 2574 scope.go:117] "RemoveContainer" containerID="27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2" Jul 20 23:30:34.410062 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.410044 2574 scope.go:117] "RemoveContainer" containerID="3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247" Jul 20 23:30:34.418215 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.418197 2574 scope.go:117] "RemoveContainer" containerID="27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2" Jul 20 23:30:34.418495 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:30:34.418475 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2\": container with ID starting with 27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2 not found: ID does not exist" containerID="27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2" Jul 20 23:30:34.418549 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.418504 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2"} err="failed to get container status \"27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2\": rpc error: code = NotFound desc = could not find container \"27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2\": container with ID starting with 27c449230c2c909badac1a461e182196140d839920b125d9742680db3c6aedb2 not found: ID does not exist" Jul 20 23:30:34.418549 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.418523 2574 scope.go:117] "RemoveContainer" containerID="3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247" Jul 20 23:30:34.418792 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:30:34.418775 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247\": container with ID starting with 3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247 not found: ID does not exist" containerID="3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247" Jul 20 23:30:34.418833 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.418798 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247"} err="failed to get container status \"3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247\": rpc error: code = NotFound desc = could not find container \"3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247\": container with ID starting with 3f5a44c4bcf741a25773d1cb433313870b1215e394cb46d38e29c071c0133247 not found: ID does not exist" Jul 20 23:30:34.421630 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.421607 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg"] Jul 20 23:30:34.427739 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:34.427717 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6a0c93-openshift-default-86dd7b45c7-vzsjg"] Jul 20 23:30:35.232774 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:35.232735 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" path="/var/lib/kubelet/pods/6f99a772-e13a-4813-8628-5bb317027aeb/volumes" Jul 20 23:30:39.439977 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.439939 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j"] Jul 20 23:30:39.440432 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.440151 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" podUID="88a2e127-740e-4f53-a82d-a3217a0412b6" containerName="maas-api" containerID="cri-o://def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae" gracePeriod=30 Jul 20 23:30:39.694638 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.694572 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:30:39.807391 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.807354 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a2e127-740e-4f53-a82d-a3217a0412b6-openshift-service-ca\") pod \"88a2e127-740e-4f53-a82d-a3217a0412b6\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " Jul 20 23:30:39.807619 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.807407 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjwss\" (UniqueName: \"kubernetes.io/projected/88a2e127-740e-4f53-a82d-a3217a0412b6-kube-api-access-qjwss\") pod \"88a2e127-740e-4f53-a82d-a3217a0412b6\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " Jul 20 23:30:39.807619 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.807432 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a2e127-740e-4f53-a82d-a3217a0412b6-maas-api-tls\") pod \"88a2e127-740e-4f53-a82d-a3217a0412b6\" (UID: \"88a2e127-740e-4f53-a82d-a3217a0412b6\") " Jul 20 23:30:39.807779 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.807754 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a2e127-740e-4f53-a82d-a3217a0412b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "88a2e127-740e-4f53-a82d-a3217a0412b6" (UID: "88a2e127-740e-4f53-a82d-a3217a0412b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:30:39.809589 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.809567 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a2e127-740e-4f53-a82d-a3217a0412b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "88a2e127-740e-4f53-a82d-a3217a0412b6" (UID: "88a2e127-740e-4f53-a82d-a3217a0412b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:30:39.809679 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.809655 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a2e127-740e-4f53-a82d-a3217a0412b6-kube-api-access-qjwss" (OuterVolumeSpecName: "kube-api-access-qjwss") pod "88a2e127-740e-4f53-a82d-a3217a0412b6" (UID: "88a2e127-740e-4f53-a82d-a3217a0412b6"). InnerVolumeSpecName "kube-api-access-qjwss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:30:39.908794 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.908756 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a2e127-740e-4f53-a82d-a3217a0412b6-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:39.908794 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.908786 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjwss\" (UniqueName: \"kubernetes.io/projected/88a2e127-740e-4f53-a82d-a3217a0412b6-kube-api-access-qjwss\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:39.908794 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:39.908797 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88a2e127-740e-4f53-a82d-a3217a0412b6-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:30:40.429195 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:40.429163 2574 generic.go:356] "Generic (PLEG): container finished" podID="88a2e127-740e-4f53-a82d-a3217a0412b6" containerID="def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae" exitCode=0 Jul 20 23:30:40.429371 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:40.429207 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" Jul 20 23:30:40.429371 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:40.429241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" event={"ID":"88a2e127-740e-4f53-a82d-a3217a0412b6","Type":"ContainerDied","Data":"def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae"} Jul 20 23:30:40.429371 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:40.429293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j" event={"ID":"88a2e127-740e-4f53-a82d-a3217a0412b6","Type":"ContainerDied","Data":"37f23b19daa5a58a3e65131a08cd50ed35c029c813b572f2837df7fead6eefcb"} Jul 20 23:30:40.429371 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:40.429316 2574 scope.go:117] "RemoveContainer" containerID="def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae" Jul 20 23:30:40.438879 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:40.438859 2574 scope.go:117] "RemoveContainer" containerID="def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae" Jul 20 23:30:40.439140 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:30:40.439122 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae\": container with ID starting with def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae not found: ID does not exist" containerID="def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae" Jul 20 23:30:40.439213 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:40.439153 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae"} err="failed to get container status \"def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae\": rpc error: code = NotFound desc = could not find container \"def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae\": container with ID starting with def6a0c39d9b5b4128d22c7ce4f8a1aa4c1c15885858d5a77db73f479995edae not found: ID does not exist" Jul 20 23:30:40.452452 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:40.452426 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j"] Jul 20 23:30:40.455490 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:40.455452 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-324c54-7469965d5b-j2k6j"] Jul 20 23:30:41.227325 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:41.227289 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a2e127-740e-4f53-a82d-a3217a0412b6" path="/var/lib/kubelet/pods/88a2e127-740e-4f53-a82d-a3217a0412b6/volumes" Jul 20 23:30:51.458605 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.458568 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf"] Jul 20 23:30:51.459180 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459146 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" Jul 20 23:30:51.459327 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459302 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" Jul 20 23:30:51.459449 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459435 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" Jul 20 23:30:51.459588 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459566 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" Jul 20 23:30:51.459588 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459601 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124c0efa-d056-4891-b18b-aa6762c741d8" containerName="maas-api" Jul 20 23:30:51.459750 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459610 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="124c0efa-d056-4891-b18b-aa6762c741d8" containerName="maas-api" Jul 20 23:30:51.459750 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459623 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db302c56-2c83-4b55-aa95-200c16138598" containerName="main" Jul 20 23:30:51.459750 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459631 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db302c56-2c83-4b55-aa95-200c16138598" containerName="main" Jul 20 23:30:51.459750 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459648 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a2e127-740e-4f53-a82d-a3217a0412b6" containerName="maas-api" Jul 20 23:30:51.459750 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459655 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a2e127-740e-4f53-a82d-a3217a0412b6" containerName="maas-api" Jul 20 23:30:51.459935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459767 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" Jul 20 23:30:51.459935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459778 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="db302c56-2c83-4b55-aa95-200c16138598" containerName="main" Jul 20 23:30:51.459935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459791 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f99a772-e13a-4813-8628-5bb317027aeb" containerName="istio-proxy" Jul 20 23:30:51.459935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459804 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a2e127-740e-4f53-a82d-a3217a0412b6" containerName="maas-api" Jul 20 23:30:51.459935 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.459811 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="124c0efa-d056-4891-b18b-aa6762c741d8" containerName="maas-api" Jul 20 23:30:51.462815 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.462796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.465551 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.465524 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c4462e-openshift-default-dockercfg-vrfj2\"" Jul 20 23:30:51.473659 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.473634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf"] Jul 20 23:30:51.616284 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.616240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-certs\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.616284 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.616284 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f383c625-5221-47ed-ab2e-ddbb2d204aef-istiod-ca-cert\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.616590 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.616302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-socket\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.616590 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.616329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwg7\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-kube-api-access-szwg7\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.616590 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.616391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-credential-socket\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.616590 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.616425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-token\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.616590 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.616491 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-envoy\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.616590 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.616507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-podinfo\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.616590 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.616544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-data\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-credential-socket\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-token\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-envoy\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-podinfo\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717504 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-data\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717504 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-certs\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717504 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717256 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f383c625-5221-47ed-ab2e-ddbb2d204aef-istiod-ca-cert\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717504 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-socket\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717504 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717388 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwg7\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-kube-api-access-szwg7\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717785 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-credential-socket\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717785 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717607 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-data\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717785 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-certs\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.717905 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.717780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-socket\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.718205 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.718185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f383c625-5221-47ed-ab2e-ddbb2d204aef-istiod-ca-cert\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.719629 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.719609 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-envoy\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.719840 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.719824 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-podinfo\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.725740 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.725713 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwg7\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-kube-api-access-szwg7\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.725880 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.725862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-token\") pod \"e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.773640 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.773599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:51.912412 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.912373 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf"] Jul 20 23:30:51.916262 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:30:51.916230 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf383c625_5221_47ed_ab2e_ddbb2d204aef.slice/crio-90d5a7a6bc2a98a2c1d75c9cb65307604c33a2b20aab3f05a59c887650c72bc0 WatchSource:0}: Error finding container 90d5a7a6bc2a98a2c1d75c9cb65307604c33a2b20aab3f05a59c887650c72bc0: Status 404 returned error can't find the container with id 90d5a7a6bc2a98a2c1d75c9cb65307604c33a2b20aab3f05a59c887650c72bc0 Jul 20 23:30:51.918208 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.918174 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:30:51.918298 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.918245 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:30:51.918298 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:51.918274 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:30:52.494567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:52.494530 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" event={"ID":"f383c625-5221-47ed-ab2e-ddbb2d204aef","Type":"ContainerStarted","Data":"68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a"} Jul 20 23:30:52.494966 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:52.494598 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" event={"ID":"f383c625-5221-47ed-ab2e-ddbb2d204aef","Type":"ContainerStarted","Data":"90d5a7a6bc2a98a2c1d75c9cb65307604c33a2b20aab3f05a59c887650c72bc0"} Jul 20 23:30:52.516822 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:52.516740 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" podStartSLOduration=1.516720305 podStartE2EDuration="1.516720305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:30:52.513320699 +0000 UTC m=+2227.831672985" watchObservedRunningTime="2026-07-20 23:30:52.516720305 +0000 UTC m=+2227.835072592" Jul 20 23:30:52.774741 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:52.774654 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:52.779742 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:52.779714 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:53.498937 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:53.498904 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:53.499909 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:53.499891 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:30:54.469518 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.469490 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr"] Jul 20 23:30:54.472623 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.472603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:54.475506 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.475481 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c4462e-serving-cert\"" Jul 20 23:30:54.485782 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.485748 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr"] Jul 20 23:30:54.643509 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.643440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6gl\" (UniqueName: \"kubernetes.io/projected/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-kube-api-access-zb6gl\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:54.643915 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.643538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-maas-api-tls\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:54.643915 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.643644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-openshift-service-ca\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:54.745058 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.744965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-maas-api-tls\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:54.745058 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.745026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-openshift-service-ca\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:54.745276 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:30:54.745117 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-serving-cert: secret "maas-api-e2e-api-a-c4462e-serving-cert" not found Jul 20 23:30:54.745276 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.745185 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6gl\" (UniqueName: \"kubernetes.io/projected/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-kube-api-access-zb6gl\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:54.745276 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:30:54.745198 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-maas-api-tls podName:972f19b0-eaaf-4254-bb08-f8f3ef7922fd nodeName:}" failed. No retries permitted until 2026-07-20 23:30:55.245176042 +0000 UTC m=+2230.563528317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-maas-api-tls") pod "maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" (UID: "972f19b0-eaaf-4254-bb08-f8f3ef7922fd") : secret "maas-api-e2e-api-a-c4462e-serving-cert" not found Jul 20 23:30:54.745699 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.745682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-openshift-service-ca\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:54.753922 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:54.753899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6gl\" (UniqueName: \"kubernetes.io/projected/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-kube-api-access-zb6gl\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:55.250208 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:55.250173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-maas-api-tls\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:55.252815 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:55.252791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-maas-api-tls\") pod \"maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:55.385662 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:55.385626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:55.534838 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:55.534804 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr"] Jul 20 23:30:55.536057 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:30:55.536026 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972f19b0_eaaf_4254_bb08_f8f3ef7922fd.slice/crio-02cca600e17d27046aa49401f4d2f879377cd02d499208a88665031d2fe81b81 WatchSource:0}: Error finding container 02cca600e17d27046aa49401f4d2f879377cd02d499208a88665031d2fe81b81: Status 404 returned error can't find the container with id 02cca600e17d27046aa49401f4d2f879377cd02d499208a88665031d2fe81b81 Jul 20 23:30:55.537509 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:55.537489 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 23:30:56.516069 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:56.516032 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" event={"ID":"972f19b0-eaaf-4254-bb08-f8f3ef7922fd","Type":"ContainerStarted","Data":"8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7"} Jul 20 23:30:56.516489 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:56.516085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" event={"ID":"972f19b0-eaaf-4254-bb08-f8f3ef7922fd","Type":"ContainerStarted","Data":"02cca600e17d27046aa49401f4d2f879377cd02d499208a88665031d2fe81b81"} Jul 20 23:30:56.516489 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:56.516111 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:30:56.536766 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:30:56.536713 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" podStartSLOduration=2.130844135 podStartE2EDuration="2.536672701s" totalImagesPullingTime="405.828566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:30:56.535589741 +0000 UTC m=+2231.853942051" watchObservedRunningTime="2026-07-20 23:30:56.536672701 +0000 UTC m=+2231.855025020" Jul 20 23:31:02.526258 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.526222 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:31:02.651484 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.651426 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr"] Jul 20 23:31:02.655778 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.655750 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:02.658715 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.658687 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4c2828-serving-cert\"" Jul 20 23:31:02.670524 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.670492 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr"] Jul 20 23:31:02.715472 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.715408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74b0c5-962f-402c-86ce-df274cec17fe-openshift-service-ca\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:02.715656 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.715577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:02.715656 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.715604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjm7k\" (UniqueName: \"kubernetes.io/projected/8a74b0c5-962f-402c-86ce-df274cec17fe-kube-api-access-pjm7k\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:02.816179 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.816086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:02.816179 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.816131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjm7k\" (UniqueName: \"kubernetes.io/projected/8a74b0c5-962f-402c-86ce-df274cec17fe-kube-api-access-pjm7k\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:02.816179 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.816175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74b0c5-962f-402c-86ce-df274cec17fe-openshift-service-ca\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:02.816440 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:31:02.816250 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-serving-cert: secret "maas-api-e2e-api-b-4c2828-serving-cert" not found Jul 20 23:31:02.816440 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:31:02.816342 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls podName:8a74b0c5-962f-402c-86ce-df274cec17fe nodeName:}" failed. No retries permitted until 2026-07-20 23:31:03.316324127 +0000 UTC m=+2238.634676390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls") pod "maas-api-e2e-api-b-4c2828-597465f89c-6pffr" (UID: "8a74b0c5-962f-402c-86ce-df274cec17fe") : secret "maas-api-e2e-api-b-4c2828-serving-cert" not found Jul 20 23:31:02.816872 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.816855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74b0c5-962f-402c-86ce-df274cec17fe-openshift-service-ca\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:02.824966 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:02.824933 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjm7k\" (UniqueName: \"kubernetes.io/projected/8a74b0c5-962f-402c-86ce-df274cec17fe-kube-api-access-pjm7k\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:03.319764 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:03.319722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:03.319991 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:31:03.319889 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-serving-cert: secret "maas-api-e2e-api-b-4c2828-serving-cert" not found Jul 20 23:31:03.319991 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:31:03.319968 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls podName:8a74b0c5-962f-402c-86ce-df274cec17fe nodeName:}" failed. No retries permitted until 2026-07-20 23:31:04.319948634 +0000 UTC m=+2239.638300914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls") pod "maas-api-e2e-api-b-4c2828-597465f89c-6pffr" (UID: "8a74b0c5-962f-402c-86ce-df274cec17fe") : secret "maas-api-e2e-api-b-4c2828-serving-cert" not found Jul 20 23:31:04.326568 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:04.326533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:04.329095 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:04.329067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls\") pod \"maas-api-e2e-api-b-4c2828-597465f89c-6pffr\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:04.487447 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:04.487408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:04.644346 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:04.644320 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr"] Jul 20 23:31:04.646170 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:31:04.646142 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a74b0c5_962f_402c_86ce_df274cec17fe.slice/crio-16f048e6867646f33ecfe473d74239c7003ed60c424fb4c9c6cee1107bc4a121 WatchSource:0}: Error finding container 16f048e6867646f33ecfe473d74239c7003ed60c424fb4c9c6cee1107bc4a121: Status 404 returned error can't find the container with id 16f048e6867646f33ecfe473d74239c7003ed60c424fb4c9c6cee1107bc4a121 Jul 20 23:31:05.562721 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:05.562676 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" event={"ID":"8a74b0c5-962f-402c-86ce-df274cec17fe","Type":"ContainerStarted","Data":"0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657"} Jul 20 23:31:05.563221 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:05.562777 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" event={"ID":"8a74b0c5-962f-402c-86ce-df274cec17fe","Type":"ContainerStarted","Data":"16f048e6867646f33ecfe473d74239c7003ed60c424fb4c9c6cee1107bc4a121"} Jul 20 23:31:05.563221 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:05.562824 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:09.090515 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:09.089672 2574 patch_prober.go:28] interesting pod/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:33900->10.133.0.59:15021: read: connection reset by peer" start-of-body= Jul 20 23:31:09.090515 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:09.089755 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:33900->10.133.0.59:15021: read: connection reset by peer" Jul 20 23:31:09.585486 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:09.585423 2574 generic.go:356] "Generic (PLEG): container finished" podID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerID="68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a" exitCode=0 Jul 20 23:31:09.585678 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:09.585501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" event={"ID":"f383c625-5221-47ed-ab2e-ddbb2d204aef","Type":"ContainerDied","Data":"68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a"} Jul 20 23:31:09.585884 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:09.585869 2574 scope.go:117] "RemoveContainer" containerID="68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a" Jul 20 23:31:09.586850 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:09.586818 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:31:09.586948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:09.586917 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:31:09.587016 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:09.586975 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:31:09.607544 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:09.607494 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" podStartSLOduration=7.206862235 podStartE2EDuration="7.607475053s" totalImagesPullingTime="400.612818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:31:05.582448952 +0000 UTC m=+2240.900801240" watchObservedRunningTime="2026-07-20 23:31:09.607475053 +0000 UTC m=+2244.925827341" Jul 20 23:31:10.592533 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:10.592499 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" event={"ID":"f383c625-5221-47ed-ab2e-ddbb2d204aef","Type":"ContainerStarted","Data":"6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce"} Jul 20 23:31:10.774476 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:10.774411 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:31:10.774924 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:10.774891 2574 patch_prober.go:28] interesting pod/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 20 23:31:10.775037 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:10.774953 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 20 23:31:11.581824 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:11.581773 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:11.774379 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:11.774336 2574 patch_prober.go:28] interesting pod/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 20 23:31:11.775039 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:11.774999 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 20 23:31:12.774546 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:12.774507 2574 patch_prober.go:28] interesting pod/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 20 23:31:12.775138 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:12.775108 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 20 23:31:13.774754 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:13.774702 2574 patch_prober.go:28] interesting pod/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 20 23:31:13.775255 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:13.774778 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 20 23:31:14.774995 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:14.774514 2574 patch_prober.go:28] interesting pod/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 20 23:31:14.774995 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:14.774594 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 20 23:31:15.777380 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:15.777345 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:31:15.777846 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:15.777650 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:31:15.778358 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:15.778342 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:31:29.652131 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:29.652092 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr"] Jul 20 23:31:29.652737 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:29.652409 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" podUID="972f19b0-eaaf-4254-bb08-f8f3ef7922fd" containerName="maas-api" containerID="cri-o://8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7" gracePeriod=30 Jul 20 23:31:29.905231 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:29.905158 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:31:29.955436 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:29.955403 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb6gl\" (UniqueName: \"kubernetes.io/projected/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-kube-api-access-zb6gl\") pod \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " Jul 20 23:31:29.955650 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:29.955512 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-maas-api-tls\") pod \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " Jul 20 23:31:29.955650 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:29.955637 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-openshift-service-ca\") pod \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\" (UID: \"972f19b0-eaaf-4254-bb08-f8f3ef7922fd\") " Jul 20 23:31:29.955963 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:29.955941 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "972f19b0-eaaf-4254-bb08-f8f3ef7922fd" (UID: "972f19b0-eaaf-4254-bb08-f8f3ef7922fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:31:29.957680 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:29.957651 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "972f19b0-eaaf-4254-bb08-f8f3ef7922fd" (UID: "972f19b0-eaaf-4254-bb08-f8f3ef7922fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:31:29.957791 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:29.957683 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-kube-api-access-zb6gl" (OuterVolumeSpecName: "kube-api-access-zb6gl") pod "972f19b0-eaaf-4254-bb08-f8f3ef7922fd" (UID: "972f19b0-eaaf-4254-bb08-f8f3ef7922fd"). InnerVolumeSpecName "kube-api-access-zb6gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:31:30.057111 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.057073 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:30.057111 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.057107 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb6gl\" (UniqueName: \"kubernetes.io/projected/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-kube-api-access-zb6gl\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:30.057111 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.057116 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972f19b0-eaaf-4254-bb08-f8f3ef7922fd-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:30.689296 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.689258 2574 generic.go:356] "Generic (PLEG): container finished" podID="972f19b0-eaaf-4254-bb08-f8f3ef7922fd" containerID="8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7" exitCode=0 Jul 20 23:31:30.689848 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.689312 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" Jul 20 23:31:30.689848 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.689331 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" event={"ID":"972f19b0-eaaf-4254-bb08-f8f3ef7922fd","Type":"ContainerDied","Data":"8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7"} Jul 20 23:31:30.689848 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.689383 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr" event={"ID":"972f19b0-eaaf-4254-bb08-f8f3ef7922fd","Type":"ContainerDied","Data":"02cca600e17d27046aa49401f4d2f879377cd02d499208a88665031d2fe81b81"} Jul 20 23:31:30.689848 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.689414 2574 scope.go:117] "RemoveContainer" containerID="8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7" Jul 20 23:31:30.698537 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.698517 2574 scope.go:117] "RemoveContainer" containerID="8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7" Jul 20 23:31:30.698863 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:31:30.698842 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7\": container with ID starting with 8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7 not found: ID does not exist" containerID="8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7" Jul 20 23:31:30.698918 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.698873 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7"} err="failed to get container status \"8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7\": rpc error: code = NotFound desc = could not find container \"8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7\": container with ID starting with 8f8f6e8536bf39af7000c237b1495f3e29e8d2d74f5694f34461299929c1c5d7 not found: ID does not exist" Jul 20 23:31:30.711506 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.711478 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr"] Jul 20 23:31:30.715093 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:30.715072 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4462e-6d45c744b7-8s9jr"] Jul 20 23:31:31.228468 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:31.228426 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972f19b0-eaaf-4254-bb08-f8f3ef7922fd" path="/var/lib/kubelet/pods/972f19b0-eaaf-4254-bb08-f8f3ef7922fd/volumes" Jul 20 23:31:40.977600 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:40.977560 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf"] Jul 20 23:31:40.978101 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:40.977828 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" containerID="cri-o://6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce" gracePeriod=30 Jul 20 23:31:45.780580 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:45.780542 2574 patch_prober.go:28] interesting pod/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 20 23:31:45.781140 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:45.780620 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 23:31:46.232279 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.232252 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:31:46.397466 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397437 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-envoy\") pod \"f383c625-5221-47ed-ab2e-ddbb2d204aef\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " Jul 20 23:31:46.397651 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397541 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-credential-socket\") pod \"f383c625-5221-47ed-ab2e-ddbb2d204aef\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " Jul 20 23:31:46.397651 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397577 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f383c625-5221-47ed-ab2e-ddbb2d204aef-istiod-ca-cert\") pod \"f383c625-5221-47ed-ab2e-ddbb2d204aef\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " Jul 20 23:31:46.397651 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397609 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-podinfo\") pod \"f383c625-5221-47ed-ab2e-ddbb2d204aef\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " Jul 20 23:31:46.397651 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397640 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-socket\") pod \"f383c625-5221-47ed-ab2e-ddbb2d204aef\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " Jul 20 23:31:46.397855 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397693 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-data\") pod \"f383c625-5221-47ed-ab2e-ddbb2d204aef\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " Jul 20 23:31:46.397855 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397745 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szwg7\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-kube-api-access-szwg7\") pod \"f383c625-5221-47ed-ab2e-ddbb2d204aef\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " Jul 20 23:31:46.397855 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397778 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-certs\") pod \"f383c625-5221-47ed-ab2e-ddbb2d204aef\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " Jul 20 23:31:46.397855 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397799 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-token\") pod \"f383c625-5221-47ed-ab2e-ddbb2d204aef\" (UID: \"f383c625-5221-47ed-ab2e-ddbb2d204aef\") " Jul 20 23:31:46.398053 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397891 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f383c625-5221-47ed-ab2e-ddbb2d204aef" (UID: "f383c625-5221-47ed-ab2e-ddbb2d204aef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:31:46.398053 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397980 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f383c625-5221-47ed-ab2e-ddbb2d204aef" (UID: "f383c625-5221-47ed-ab2e-ddbb2d204aef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:31:46.398053 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.397990 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f383c625-5221-47ed-ab2e-ddbb2d204aef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f383c625-5221-47ed-ab2e-ddbb2d204aef" (UID: "f383c625-5221-47ed-ab2e-ddbb2d204aef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:31:46.398185 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.398101 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-credential-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:46.398185 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.398123 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f383c625-5221-47ed-ab2e-ddbb2d204aef-istiod-ca-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:46.398185 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.398138 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:46.398335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.398233 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f383c625-5221-47ed-ab2e-ddbb2d204aef" (UID: "f383c625-5221-47ed-ab2e-ddbb2d204aef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:31:46.398745 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.398720 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-data" (OuterVolumeSpecName: "istio-data") pod "f383c625-5221-47ed-ab2e-ddbb2d204aef" (UID: "f383c625-5221-47ed-ab2e-ddbb2d204aef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:31:46.399957 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.399930 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-token" (OuterVolumeSpecName: "istio-token") pod "f383c625-5221-47ed-ab2e-ddbb2d204aef" (UID: "f383c625-5221-47ed-ab2e-ddbb2d204aef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:31:46.400044 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.399953 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-kube-api-access-szwg7" (OuterVolumeSpecName: "kube-api-access-szwg7") pod "f383c625-5221-47ed-ab2e-ddbb2d204aef" (UID: "f383c625-5221-47ed-ab2e-ddbb2d204aef"). InnerVolumeSpecName "kube-api-access-szwg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:31:46.400212 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.400185 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f383c625-5221-47ed-ab2e-ddbb2d204aef" (UID: "f383c625-5221-47ed-ab2e-ddbb2d204aef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:31:46.400212 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.400201 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f383c625-5221-47ed-ab2e-ddbb2d204aef" (UID: "f383c625-5221-47ed-ab2e-ddbb2d204aef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 23:31:46.499567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.499524 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-data\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:46.499567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.499557 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwg7\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-kube-api-access-szwg7\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:46.499567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.499567 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-workload-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:46.499567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.499575 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-token\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:46.499840 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.499584 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-envoy\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:46.499840 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.499591 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f383c625-5221-47ed-ab2e-ddbb2d204aef-istio-podinfo\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:46.765016 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.764934 2574 generic.go:356] "Generic (PLEG): container finished" podID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerID="6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce" exitCode=0 Jul 20 23:31:46.765016 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.764981 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" Jul 20 23:31:46.765192 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.765018 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" event={"ID":"f383c625-5221-47ed-ab2e-ddbb2d204aef","Type":"ContainerDied","Data":"6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce"} Jul 20 23:31:46.765192 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.765074 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf" event={"ID":"f383c625-5221-47ed-ab2e-ddbb2d204aef","Type":"ContainerDied","Data":"90d5a7a6bc2a98a2c1d75c9cb65307604c33a2b20aab3f05a59c887650c72bc0"} Jul 20 23:31:46.765192 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.765098 2574 scope.go:117] "RemoveContainer" containerID="6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce" Jul 20 23:31:46.774372 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.774351 2574 scope.go:117] "RemoveContainer" containerID="68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a" Jul 20 23:31:46.782089 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.782067 2574 scope.go:117] "RemoveContainer" containerID="6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce" Jul 20 23:31:46.782433 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:31:46.782394 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce\": container with ID starting with 6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce not found: ID does not exist" containerID="6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce" Jul 20 23:31:46.782433 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.782421 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce"} err="failed to get container status \"6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce\": rpc error: code = NotFound desc = could not find container \"6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce\": container with ID starting with 6af9f06f744904660d069cbe04286bbef2449279dd103375aa5911658570afce not found: ID does not exist" Jul 20 23:31:46.782651 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.782439 2574 scope.go:117] "RemoveContainer" containerID="68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a" Jul 20 23:31:46.782719 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:31:46.782700 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a\": container with ID starting with 68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a not found: ID does not exist" containerID="68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a" Jul 20 23:31:46.782758 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.782725 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a"} err="failed to get container status \"68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a\": rpc error: code = NotFound desc = could not find container \"68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a\": container with ID starting with 68012ad1849d5778e80b2ce1776c5bd877803e820c56adc3001c928640d0239a not found: ID does not exist" Jul 20 23:31:46.787030 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.787007 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf"] Jul 20 23:31:46.791609 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:46.791586 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c4462e-openshift-default-849768f65f-gn5lf"] Jul 20 23:31:47.228229 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:47.228197 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" path="/var/lib/kubelet/pods/f383c625-5221-47ed-ab2e-ddbb2d204aef/volumes" Jul 20 23:31:56.708609 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:56.708566 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr"] Jul 20 23:31:56.709100 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:56.708829 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" podUID="8a74b0c5-962f-402c-86ce-df274cec17fe" containerName="maas-api" containerID="cri-o://0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657" gracePeriod=30 Jul 20 23:31:56.962440 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:56.962376 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:57.094705 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.094666 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjm7k\" (UniqueName: \"kubernetes.io/projected/8a74b0c5-962f-402c-86ce-df274cec17fe-kube-api-access-pjm7k\") pod \"8a74b0c5-962f-402c-86ce-df274cec17fe\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " Jul 20 23:31:57.094895 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.094776 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74b0c5-962f-402c-86ce-df274cec17fe-openshift-service-ca\") pod \"8a74b0c5-962f-402c-86ce-df274cec17fe\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " Jul 20 23:31:57.094895 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.094799 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls\") pod \"8a74b0c5-962f-402c-86ce-df274cec17fe\" (UID: \"8a74b0c5-962f-402c-86ce-df274cec17fe\") " Jul 20 23:31:57.095144 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.095111 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a74b0c5-962f-402c-86ce-df274cec17fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a74b0c5-962f-402c-86ce-df274cec17fe" (UID: "8a74b0c5-962f-402c-86ce-df274cec17fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:31:57.096832 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.096809 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a74b0c5-962f-402c-86ce-df274cec17fe-kube-api-access-pjm7k" (OuterVolumeSpecName: "kube-api-access-pjm7k") pod "8a74b0c5-962f-402c-86ce-df274cec17fe" (UID: "8a74b0c5-962f-402c-86ce-df274cec17fe"). InnerVolumeSpecName "kube-api-access-pjm7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:31:57.096908 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.096826 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a74b0c5-962f-402c-86ce-df274cec17fe" (UID: "8a74b0c5-962f-402c-86ce-df274cec17fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:31:57.196232 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.196186 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a74b0c5-962f-402c-86ce-df274cec17fe-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:57.196232 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.196233 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a74b0c5-962f-402c-86ce-df274cec17fe-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:57.196446 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.196246 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjm7k\" (UniqueName: \"kubernetes.io/projected/8a74b0c5-962f-402c-86ce-df274cec17fe-kube-api-access-pjm7k\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:31:57.817214 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.817178 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a74b0c5-962f-402c-86ce-df274cec17fe" containerID="0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657" exitCode=0 Jul 20 23:31:57.817716 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.817244 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" Jul 20 23:31:57.817716 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.817264 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" event={"ID":"8a74b0c5-962f-402c-86ce-df274cec17fe","Type":"ContainerDied","Data":"0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657"} Jul 20 23:31:57.817716 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.817319 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr" event={"ID":"8a74b0c5-962f-402c-86ce-df274cec17fe","Type":"ContainerDied","Data":"16f048e6867646f33ecfe473d74239c7003ed60c424fb4c9c6cee1107bc4a121"} Jul 20 23:31:57.817716 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.817341 2574 scope.go:117] "RemoveContainer" containerID="0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657" Jul 20 23:31:57.825659 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.825638 2574 scope.go:117] "RemoveContainer" containerID="0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657" Jul 20 23:31:57.825902 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:31:57.825883 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657\": container with ID starting with 0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657 not found: ID does not exist" containerID="0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657" Jul 20 23:31:57.825967 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.825926 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657"} err="failed to get container status \"0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657\": rpc error: code = NotFound desc = could not find container \"0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657\": container with ID starting with 0a57792be190dec916510e3e15d59774d1eacb5e12346219020ef52b297a2657 not found: ID does not exist" Jul 20 23:31:57.833914 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.833880 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr"] Jul 20 23:31:57.837037 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:57.837017 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4c2828-597465f89c-6pffr"] Jul 20 23:31:59.227749 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:31:59.227709 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a74b0c5-962f-402c-86ce-df274cec17fe" path="/var/lib/kubelet/pods/8a74b0c5-962f-402c-86ce-df274cec17fe/volumes" Jul 20 23:32:08.691696 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.691664 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5"] Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692166 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972f19b0-eaaf-4254-bb08-f8f3ef7922fd" containerName="maas-api" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692187 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="972f19b0-eaaf-4254-bb08-f8f3ef7922fd" containerName="maas-api" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692199 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692207 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692224 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692234 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692250 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a74b0c5-962f-402c-86ce-df274cec17fe" containerName="maas-api" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692257 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a74b0c5-962f-402c-86ce-df274cec17fe" containerName="maas-api" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692343 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692356 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="972f19b0-eaaf-4254-bb08-f8f3ef7922fd" containerName="maas-api" Jul 20 23:32:08.692441 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.692366 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a74b0c5-962f-402c-86ce-df274cec17fe" containerName="maas-api" Jul 20 23:32:08.696641 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.696623 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.700807 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.700790 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-107ae7-openshift-default-dockercfg-78224\"" Jul 20 23:32:08.711530 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.711495 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5"] Jul 20 23:32:08.783554 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.783516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-credential-socket\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.783731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.783564 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b99k\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-kube-api-access-4b99k\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.783731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.783586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-socket\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.783731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.783659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0917a360-a964-479b-b304-3b49069bcdaf-istio-podinfo\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.783731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.783692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-istio-token\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.783872 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.783736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-certs\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.783872 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.783770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-envoy\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.783872 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.783785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-data\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.783872 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.783841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0917a360-a964-479b-b304-3b49069bcdaf-istiod-ca-cert\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.884667 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.884633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-credential-socket\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.884865 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.884693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b99k\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-kube-api-access-4b99k\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.884865 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.884713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-socket\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.884865 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.884747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0917a360-a964-479b-b304-3b49069bcdaf-istio-podinfo\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885038 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.884926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-istio-token\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885038 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.884979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-certs\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885038 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.885003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-envoy\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885203 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.885035 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-credential-socket\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885203 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.885045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-data\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885203 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.885104 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-socket\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885203 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.885109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0917a360-a964-479b-b304-3b49069bcdaf-istiod-ca-cert\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885416 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.885263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-certs\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885506 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.885418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-data\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.885810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.885789 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0917a360-a964-479b-b304-3b49069bcdaf-istiod-ca-cert\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.887076 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.887056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-envoy\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.887281 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.887264 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0917a360-a964-479b-b304-3b49069bcdaf-istio-podinfo\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.892976 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.892953 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-istio-token\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:08.893093 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:08.893076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b99k\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-kube-api-access-4b99k\") pod \"e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:09.007125 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:09.007035 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:09.165323 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:09.165295 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5"] Jul 20 23:32:09.166705 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:32:09.166675 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0917a360_a964_479b_b304_3b49069bcdaf.slice/crio-5c40e58ee639aa0379620c6d6e153e190826cd46d184dd43b7216b2f761f6777 WatchSource:0}: Error finding container 5c40e58ee639aa0379620c6d6e153e190826cd46d184dd43b7216b2f761f6777: Status 404 returned error can't find the container with id 5c40e58ee639aa0379620c6d6e153e190826cd46d184dd43b7216b2f761f6777 Jul 20 23:32:09.168834 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:09.168790 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:32:09.168941 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:09.168871 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:32:09.169005 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:09.168944 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:32:09.874892 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:09.874851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" event={"ID":"0917a360-a964-479b-b304-3b49069bcdaf","Type":"ContainerStarted","Data":"546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646"} Jul 20 23:32:09.875274 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:09.874908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" event={"ID":"0917a360-a964-479b-b304-3b49069bcdaf","Type":"ContainerStarted","Data":"5c40e58ee639aa0379620c6d6e153e190826cd46d184dd43b7216b2f761f6777"} Jul 20 23:32:09.896098 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:09.896021 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" podStartSLOduration=1.896004965 podStartE2EDuration="1.896004965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:32:09.895186666 +0000 UTC m=+2305.213538954" watchObservedRunningTime="2026-07-20 23:32:09.896004965 +0000 UTC m=+2305.214357251" Jul 20 23:32:10.007956 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:10.007922 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:10.013554 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:10.013524 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:10.879171 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:10.879133 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:10.880188 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:10.880167 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:11.862427 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:11.862388 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4"] Jul 20 23:32:11.863086 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:11.863065 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f383c625-5221-47ed-ab2e-ddbb2d204aef" containerName="istio-proxy" Jul 20 23:32:11.866614 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:11.866591 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:11.869956 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:11.869934 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-107ae7-serving-cert\"" Jul 20 23:32:11.876530 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:11.876506 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4"] Jul 20 23:32:11.911605 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:11.911567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vdz8\" (UniqueName: \"kubernetes.io/projected/86003d1f-0a06-4d68-b383-d280f75d8cc2-kube-api-access-7vdz8\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:11.912070 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:11.911648 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86003d1f-0a06-4d68-b383-d280f75d8cc2-maas-api-tls\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:11.912070 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:11.911779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86003d1f-0a06-4d68-b383-d280f75d8cc2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:12.012677 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:12.012645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86003d1f-0a06-4d68-b383-d280f75d8cc2-maas-api-tls\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:12.012868 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:12.012698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86003d1f-0a06-4d68-b383-d280f75d8cc2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:12.012868 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:12.012745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vdz8\" (UniqueName: \"kubernetes.io/projected/86003d1f-0a06-4d68-b383-d280f75d8cc2-kube-api-access-7vdz8\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:12.012868 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:32:12.012804 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-serving-cert: secret "maas-api-e2e-shared-a-107ae7-serving-cert" not found Jul 20 23:32:12.013028 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:32:12.012897 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86003d1f-0a06-4d68-b383-d280f75d8cc2-maas-api-tls podName:86003d1f-0a06-4d68-b383-d280f75d8cc2 nodeName:}" failed. No retries permitted until 2026-07-20 23:32:12.512881426 +0000 UTC m=+2307.831233689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86003d1f-0a06-4d68-b383-d280f75d8cc2-maas-api-tls") pod "maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" (UID: "86003d1f-0a06-4d68-b383-d280f75d8cc2") : secret "maas-api-e2e-shared-a-107ae7-serving-cert" not found Jul 20 23:32:12.013372 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:12.013354 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86003d1f-0a06-4d68-b383-d280f75d8cc2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:12.032215 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:12.032184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vdz8\" (UniqueName: \"kubernetes.io/projected/86003d1f-0a06-4d68-b383-d280f75d8cc2-kube-api-access-7vdz8\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:12.518493 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:12.518441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86003d1f-0a06-4d68-b383-d280f75d8cc2-maas-api-tls\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:12.521104 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:12.521071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86003d1f-0a06-4d68-b383-d280f75d8cc2-maas-api-tls\") pod \"maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:12.778950 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:12.778859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:12.922925 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:12.922885 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4"] Jul 20 23:32:12.924826 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:32:12.924801 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86003d1f_0a06_4d68_b383_d280f75d8cc2.slice/crio-1663ae0fdb93858300e5e5d8212fe308c4f59b3e85eb31e868477e45ca362f85 WatchSource:0}: Error finding container 1663ae0fdb93858300e5e5d8212fe308c4f59b3e85eb31e868477e45ca362f85: Status 404 returned error can't find the container with id 1663ae0fdb93858300e5e5d8212fe308c4f59b3e85eb31e868477e45ca362f85 Jul 20 23:32:13.895683 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:13.895640 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" event={"ID":"86003d1f-0a06-4d68-b383-d280f75d8cc2","Type":"ContainerStarted","Data":"fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8"} Jul 20 23:32:13.895900 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:13.895700 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" event={"ID":"86003d1f-0a06-4d68-b383-d280f75d8cc2","Type":"ContainerStarted","Data":"1663ae0fdb93858300e5e5d8212fe308c4f59b3e85eb31e868477e45ca362f85"} Jul 20 23:32:13.895900 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:13.895724 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:13.913134 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:13.913073 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" podStartSLOduration=2.510426099 podStartE2EDuration="2.913055201s" totalImagesPullingTime="402.629102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:32:13.91152725 +0000 UTC m=+2309.229879536" watchObservedRunningTime="2026-07-20 23:32:13.913055201 +0000 UTC m=+2309.231407486" Jul 20 23:32:19.843433 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:19.843398 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4"] Jul 20 23:32:19.847271 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:19.847245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:19.850202 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:19.850178 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-5b9afe-serving-cert\"" Jul 20 23:32:19.866886 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:19.866856 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4"] Jul 20 23:32:19.903226 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:19.903196 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4768-4fa8-437a-8db8-719d27294423-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:19.903226 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:19.903230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd4768-4fa8-437a-8db8-719d27294423-maas-api-tls\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:19.903493 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:19.903248 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hgzg\" (UniqueName: \"kubernetes.io/projected/7bbd4768-4fa8-437a-8db8-719d27294423-kube-api-access-4hgzg\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:19.907440 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:19.907417 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:32:20.004267 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:20.004233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4768-4fa8-437a-8db8-719d27294423-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:20.004504 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:20.004280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd4768-4fa8-437a-8db8-719d27294423-maas-api-tls\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:20.004504 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:20.004303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hgzg\" (UniqueName: \"kubernetes.io/projected/7bbd4768-4fa8-437a-8db8-719d27294423-kube-api-access-4hgzg\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:20.004504 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:32:20.004449 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-serving-cert: secret "maas-api-e2e-shared-b-5b9afe-serving-cert" not found Jul 20 23:32:20.004690 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:32:20.004540 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bbd4768-4fa8-437a-8db8-719d27294423-maas-api-tls podName:7bbd4768-4fa8-437a-8db8-719d27294423 nodeName:}" failed. No retries permitted until 2026-07-20 23:32:20.504518095 +0000 UTC m=+2315.822870361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bbd4768-4fa8-437a-8db8-719d27294423-maas-api-tls") pod "maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" (UID: "7bbd4768-4fa8-437a-8db8-719d27294423") : secret "maas-api-e2e-shared-b-5b9afe-serving-cert" not found Jul 20 23:32:20.004949 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:20.004932 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4768-4fa8-437a-8db8-719d27294423-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:20.016624 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:20.016598 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hgzg\" (UniqueName: \"kubernetes.io/projected/7bbd4768-4fa8-437a-8db8-719d27294423-kube-api-access-4hgzg\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:20.509646 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:20.509611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd4768-4fa8-437a-8db8-719d27294423-maas-api-tls\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:20.512220 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:20.512191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd4768-4fa8-437a-8db8-719d27294423-maas-api-tls\") pod \"maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:20.764213 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:20.764113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:20.941093 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:20.941065 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4"] Jul 20 23:32:20.942879 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:32:20.942854 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbd4768_4fa8_437a_8db8_719d27294423.slice/crio-20c72a071a8f8bffb4ce8f032d6972e520ed03e9120527c195a2fff623ad4863 WatchSource:0}: Error finding container 20c72a071a8f8bffb4ce8f032d6972e520ed03e9120527c195a2fff623ad4863: Status 404 returned error can't find the container with id 20c72a071a8f8bffb4ce8f032d6972e520ed03e9120527c195a2fff623ad4863 Jul 20 23:32:21.939278 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:21.939243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" event={"ID":"7bbd4768-4fa8-437a-8db8-719d27294423","Type":"ContainerStarted","Data":"27dd844f911415c45c7b54d8ab1679a4e3b59f893516dabcc29303e103c28a5e"} Jul 20 23:32:21.939510 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:21.939296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" event={"ID":"7bbd4768-4fa8-437a-8db8-719d27294423","Type":"ContainerStarted","Data":"20c72a071a8f8bffb4ce8f032d6972e520ed03e9120527c195a2fff623ad4863"} Jul 20 23:32:21.939510 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:21.939368 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:24.955057 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:24.955026 2574 generic.go:356] "Generic (PLEG): container finished" podID="0917a360-a964-479b-b304-3b49069bcdaf" containerID="546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646" exitCode=0 Jul 20 23:32:24.955492 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:24.955096 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" event={"ID":"0917a360-a964-479b-b304-3b49069bcdaf","Type":"ContainerDied","Data":"546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646"} Jul 20 23:32:24.955540 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:24.955503 2574 scope.go:117] "RemoveContainer" containerID="546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646" Jul 20 23:32:24.956525 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:24.956499 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:32:24.956593 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:24.956570 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:32:24.956637 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:24.956603 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 23:32:24.978276 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:24.978226 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" podStartSLOduration=5.523858318 podStartE2EDuration="5.978209286s" totalImagesPullingTime="454.350968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:32:21.961868287 +0000 UTC m=+2317.280220575" watchObservedRunningTime="2026-07-20 23:32:24.978209286 +0000 UTC m=+2320.296561575" Jul 20 23:32:25.879528 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:25.879497 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:25.961867 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:25.961837 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" event={"ID":"0917a360-a964-479b-b304-3b49069bcdaf","Type":"ContainerStarted","Data":"26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e"} Jul 20 23:32:26.008157 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:26.008115 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:26.009472 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:26.009409 2574 patch_prober.go:28] interesting pod/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 20 23:32:26.009647 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:26.009508 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 20 23:32:27.008058 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:27.008019 2574 patch_prober.go:28] interesting pod/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 20 23:32:27.008487 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:27.008072 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 20 23:32:27.949978 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:27.949948 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:32:28.008054 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:28.008014 2574 patch_prober.go:28] interesting pod/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 20 23:32:28.008227 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:28.008089 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 20 23:32:29.007551 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:29.007514 2574 patch_prober.go:28] interesting pod/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 20 23:32:29.007773 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:29.007592 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 20 23:32:30.008062 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:30.008031 2574 patch_prober.go:28] interesting pod/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 20 23:32:30.008486 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:30.008082 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 20 23:32:31.011409 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:31.011381 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:31.011873 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:31.011674 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:31.012362 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:31.012347 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:32:45.525214 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.525181 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t"] Jul 20 23:32:45.530016 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.529996 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.533992 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.533965 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-107ae7\"/\"openshift-service-ca.crt\"" Jul 20 23:32:45.534126 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.534006 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-107ae7\"/\"auth-test-model-107ae7-kserve-self-signed-certs\"" Jul 20 23:32:45.534126 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.534058 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-107ae7\"/\"kube-root-ca.crt\"" Jul 20 23:32:45.534221 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.534144 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-107ae7\"/\"default-dockercfg-w74zq\"" Jul 20 23:32:45.540724 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.540698 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t"] Jul 20 23:32:45.557482 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.557437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04976e2-a3a9-4457-832a-8c268ab83907-tls-certs\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.557664 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.557511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-home\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.557664 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.557568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-dshm\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.557664 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.557610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-model-cache\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.557836 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.557699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkqjs\" (UniqueName: \"kubernetes.io/projected/b04976e2-a3a9-4457-832a-8c268ab83907-kube-api-access-fkqjs\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.557836 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.557724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-tmp-dir\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.658904 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.658872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04976e2-a3a9-4457-832a-8c268ab83907-tls-certs\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.659092 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.658910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-home\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.659092 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.658937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-dshm\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.659092 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.658992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-model-cache\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.659092 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.659082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkqjs\" (UniqueName: \"kubernetes.io/projected/b04976e2-a3a9-4457-832a-8c268ab83907-kube-api-access-fkqjs\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.659317 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.659110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-tmp-dir\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.659389 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.659362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-home\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.659452 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.659425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-model-cache\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.659689 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.659664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-tmp-dir\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.661422 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.661387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-dshm\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.661564 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.661499 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04976e2-a3a9-4457-832a-8c268ab83907-tls-certs\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.667267 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.667244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkqjs\" (UniqueName: \"kubernetes.io/projected/b04976e2-a3a9-4457-832a-8c268ab83907-kube-api-access-fkqjs\") pod \"auth-test-model-107ae7-kserve-76b454fff5-hwq6t\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.840691 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.840601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:45.984995 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:45.984970 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t"] Jul 20 23:32:45.987477 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:32:45.987434 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb04976e2_a3a9_4457_832a_8c268ab83907.slice/crio-c736e3eceef3bb9c1aca4ebf49b6c042e7d40fa0aa8ebd3b3ab206f41039471a WatchSource:0}: Error finding container c736e3eceef3bb9c1aca4ebf49b6c042e7d40fa0aa8ebd3b3ab206f41039471a: Status 404 returned error can't find the container with id c736e3eceef3bb9c1aca4ebf49b6c042e7d40fa0aa8ebd3b3ab206f41039471a Jul 20 23:32:46.065767 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:46.065731 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" event={"ID":"b04976e2-a3a9-4457-832a-8c268ab83907","Type":"ContainerStarted","Data":"12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a"} Jul 20 23:32:46.065933 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:46.065778 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" event={"ID":"b04976e2-a3a9-4457-832a-8c268ab83907","Type":"ContainerStarted","Data":"c736e3eceef3bb9c1aca4ebf49b6c042e7d40fa0aa8ebd3b3ab206f41039471a"} Jul 20 23:32:46.083665 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:46.083610 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" podStartSLOduration=1.08359455 podStartE2EDuration="1.08359455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:32:46.081791974 +0000 UTC m=+2341.400144259" watchObservedRunningTime="2026-07-20 23:32:46.08359455 +0000 UTC m=+2341.401946836" Jul 20 23:32:55.840959 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:55.840922 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:55.841455 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:55.840975 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:55.854146 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:55.854117 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:32:56.123190 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:32:56.123153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:33:45.233265 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:33:45.233235 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:33:45.239512 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:33:45.239488 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:36:04.499032 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:04.498994 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t"] Jul 20 23:36:14.122670 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.122626 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26"] Jul 20 23:36:14.126472 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.126433 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.129029 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.129005 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-107ae7\"/\"sub-test-model-107ae7-kserve-self-signed-certs\"" Jul 20 23:36:14.135514 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.135484 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26"] Jul 20 23:36:14.290925 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.290885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tls-certs\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.290925 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.290939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdt8\" (UniqueName: \"kubernetes.io/projected/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-kube-api-access-nzdt8\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.291186 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.291008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-home\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.291186 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.291037 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-model-cache\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.291186 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.291083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tmp-dir\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.291186 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.291129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-dshm\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.392020 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.391926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-home\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.392020 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.391966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-model-cache\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.392020 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.392005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tmp-dir\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.392297 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.392035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-dshm\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.392297 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.392085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tls-certs\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.392297 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.392131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdt8\" (UniqueName: \"kubernetes.io/projected/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-kube-api-access-nzdt8\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.392503 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.392319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-home\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.392503 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.392385 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-model-cache\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.392503 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.392452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tmp-dir\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.394469 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.394436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-dshm\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.394779 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.394759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tls-certs\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.400119 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.400092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdt8\" (UniqueName: \"kubernetes.io/projected/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-kube-api-access-nzdt8\") pod \"sub-test-model-107ae7-kserve-858f67c665-cpt26\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.436419 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.436383 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:14.570549 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:14.570522 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26"] Jul 20 23:36:14.572632 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:36:14.572598 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cd6e7db_b243_4b7a_a32a_be14abd81b5a.slice/crio-530614565d93b82f1273b29aa191b1ac045ea6f50129c3cfbdf326f64584b105 WatchSource:0}: Error finding container 530614565d93b82f1273b29aa191b1ac045ea6f50129c3cfbdf326f64584b105: Status 404 returned error can't find the container with id 530614565d93b82f1273b29aa191b1ac045ea6f50129c3cfbdf326f64584b105 Jul 20 23:36:15.065607 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:15.065566 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" event={"ID":"6cd6e7db-b243-4b7a-a32a-be14abd81b5a","Type":"ContainerStarted","Data":"022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253"} Jul 20 23:36:15.065607 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:15.065608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" event={"ID":"6cd6e7db-b243-4b7a-a32a-be14abd81b5a","Type":"ContainerStarted","Data":"530614565d93b82f1273b29aa191b1ac045ea6f50129c3cfbdf326f64584b105"} Jul 20 23:36:15.081624 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:15.081575 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" podStartSLOduration=1.081558791 podStartE2EDuration="1.081558791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:36:15.081156832 +0000 UTC m=+2550.399509123" watchObservedRunningTime="2026-07-20 23:36:15.081558791 +0000 UTC m=+2550.399911074" Jul 20 23:36:19.519637 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.519569 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" podUID="b04976e2-a3a9-4457-832a-8c268ab83907" containerName="main" containerID="cri-o://12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a" gracePeriod=45 Jul 20 23:36:19.782940 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.782876 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:36:19.943391 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943357 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-dshm\") pod \"b04976e2-a3a9-4457-832a-8c268ab83907\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " Jul 20 23:36:19.943614 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943419 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04976e2-a3a9-4457-832a-8c268ab83907-tls-certs\") pod \"b04976e2-a3a9-4457-832a-8c268ab83907\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " Jul 20 23:36:19.943614 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943494 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-tmp-dir\") pod \"b04976e2-a3a9-4457-832a-8c268ab83907\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " Jul 20 23:36:19.943614 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943518 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkqjs\" (UniqueName: \"kubernetes.io/projected/b04976e2-a3a9-4457-832a-8c268ab83907-kube-api-access-fkqjs\") pod \"b04976e2-a3a9-4457-832a-8c268ab83907\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " Jul 20 23:36:19.943614 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943589 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-home\") pod \"b04976e2-a3a9-4457-832a-8c268ab83907\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " Jul 20 23:36:19.943840 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943624 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-model-cache\") pod \"b04976e2-a3a9-4457-832a-8c268ab83907\" (UID: \"b04976e2-a3a9-4457-832a-8c268ab83907\") " Jul 20 23:36:19.943840 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943748 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b04976e2-a3a9-4457-832a-8c268ab83907" (UID: "b04976e2-a3a9-4457-832a-8c268ab83907"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:36:19.943952 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943904 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-home" (OuterVolumeSpecName: "home") pod "b04976e2-a3a9-4457-832a-8c268ab83907" (UID: "b04976e2-a3a9-4457-832a-8c268ab83907"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:36:19.943952 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943922 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-model-cache" (OuterVolumeSpecName: "model-cache") pod "b04976e2-a3a9-4457-832a-8c268ab83907" (UID: "b04976e2-a3a9-4457-832a-8c268ab83907"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:36:19.944056 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.943964 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-tmp-dir\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:36:19.945757 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.945732 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-dshm" (OuterVolumeSpecName: "dshm") pod "b04976e2-a3a9-4457-832a-8c268ab83907" (UID: "b04976e2-a3a9-4457-832a-8c268ab83907"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:36:19.945867 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.945784 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04976e2-a3a9-4457-832a-8c268ab83907-kube-api-access-fkqjs" (OuterVolumeSpecName: "kube-api-access-fkqjs") pod "b04976e2-a3a9-4457-832a-8c268ab83907" (UID: "b04976e2-a3a9-4457-832a-8c268ab83907"). InnerVolumeSpecName "kube-api-access-fkqjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:36:19.945867 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:19.945843 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04976e2-a3a9-4457-832a-8c268ab83907-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b04976e2-a3a9-4457-832a-8c268ab83907" (UID: "b04976e2-a3a9-4457-832a-8c268ab83907"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:36:20.044663 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.044575 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-home\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.044663 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.044604 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-model-cache\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.044663 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.044614 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04976e2-a3a9-4457-832a-8c268ab83907-dshm\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.044663 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.044622 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04976e2-a3a9-4457-832a-8c268ab83907-tls-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.044663 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.044630 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkqjs\" (UniqueName: \"kubernetes.io/projected/b04976e2-a3a9-4457-832a-8c268ab83907-kube-api-access-fkqjs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.090747 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.090711 2574 generic.go:356] "Generic (PLEG): container finished" podID="b04976e2-a3a9-4457-832a-8c268ab83907" containerID="12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a" exitCode=0 Jul 20 23:36:20.090747 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.090747 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" Jul 20 23:36:20.090971 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.090787 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" event={"ID":"b04976e2-a3a9-4457-832a-8c268ab83907","Type":"ContainerDied","Data":"12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a"} Jul 20 23:36:20.090971 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.090833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t" event={"ID":"b04976e2-a3a9-4457-832a-8c268ab83907","Type":"ContainerDied","Data":"c736e3eceef3bb9c1aca4ebf49b6c042e7d40fa0aa8ebd3b3ab206f41039471a"} Jul 20 23:36:20.090971 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.090854 2574 scope.go:117] "RemoveContainer" containerID="12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a" Jul 20 23:36:20.099810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.099793 2574 scope.go:117] "RemoveContainer" containerID="12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a" Jul 20 23:36:20.100071 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:36:20.100052 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a\": container with ID starting with 12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a not found: ID does not exist" containerID="12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a" Jul 20 23:36:20.100142 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.100081 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a"} err="failed to get container status \"12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a\": rpc error: code = NotFound desc = could not find container \"12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a\": container with ID starting with 12a3ec060ea0a4359b04d9118a337976396c9dc09b98107c773a10157e61923a not found: ID does not exist" Jul 20 23:36:20.111616 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.111577 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t"] Jul 20 23:36:20.115209 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:20.115188 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/auth-test-model-107ae7-kserve-76b454fff5-hwq6t"] Jul 20 23:36:21.228370 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:21.228334 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b04976e2-a3a9-4457-832a-8c268ab83907" path="/var/lib/kubelet/pods/b04976e2-a3a9-4457-832a-8c268ab83907/volumes" Jul 20 23:36:24.437127 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:24.437074 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:24.437574 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:24.437147 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:24.449782 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:24.449756 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:36:25.125455 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:36:25.125427 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:37:08.320486 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:08.320392 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26"] Jul 20 23:37:18.197271 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.197237 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd"] Jul 20 23:37:18.197798 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.197621 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04976e2-a3a9-4457-832a-8c268ab83907" containerName="main" Jul 20 23:37:18.197798 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.197632 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04976e2-a3a9-4457-832a-8c268ab83907" containerName="main" Jul 20 23:37:18.197798 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.197684 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04976e2-a3a9-4457-832a-8c268ab83907" containerName="main" Jul 20 23:37:18.200667 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.200644 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.203263 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.203238 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-107ae7\"/\"rate-test-model-107ae7-kserve-self-signed-certs\"" Jul 20 23:37:18.210205 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.210180 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd"] Jul 20 23:37:18.260719 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.260674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qcm\" (UniqueName: \"kubernetes.io/projected/c46cefdb-a599-4ed2-8985-37e63865f77d-kube-api-access-r5qcm\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.260889 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.260723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-home\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.260889 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.260783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-tmp-dir\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.260889 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.260878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-model-cache\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.261034 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.260959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46cefdb-a599-4ed2-8985-37e63865f77d-tls-certs\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.261034 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.260992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-dshm\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.364491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.362953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-model-cache\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.364491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.363034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46cefdb-a599-4ed2-8985-37e63865f77d-tls-certs\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.364491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.363079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-dshm\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.364491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.363166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qcm\" (UniqueName: \"kubernetes.io/projected/c46cefdb-a599-4ed2-8985-37e63865f77d-kube-api-access-r5qcm\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.364491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.363200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-home\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.364491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.363228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-tmp-dir\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.364491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.363865 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-tmp-dir\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.364491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.364114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-model-cache\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.367813 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.367766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46cefdb-a599-4ed2-8985-37e63865f77d-tls-certs\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.368488 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.368445 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-home\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.369973 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.369949 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-dshm\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.376675 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.376654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qcm\" (UniqueName: \"kubernetes.io/projected/c46cefdb-a599-4ed2-8985-37e63865f77d-kube-api-access-r5qcm\") pod \"rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.510393 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.510301 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:18.645765 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:18.645735 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd"] Jul 20 23:37:18.647379 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:37:18.647355 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc46cefdb_a599_4ed2_8985_37e63865f77d.slice/crio-b90b528f7e7dbfbc0c62b11d52b5df649bd8b554c09f9a221243d4de5a3c5d81 WatchSource:0}: Error finding container b90b528f7e7dbfbc0c62b11d52b5df649bd8b554c09f9a221243d4de5a3c5d81: Status 404 returned error can't find the container with id b90b528f7e7dbfbc0c62b11d52b5df649bd8b554c09f9a221243d4de5a3c5d81 Jul 20 23:37:19.377265 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:19.377233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" event={"ID":"c46cefdb-a599-4ed2-8985-37e63865f77d","Type":"ContainerStarted","Data":"37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631"} Jul 20 23:37:19.377635 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:19.377272 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" event={"ID":"c46cefdb-a599-4ed2-8985-37e63865f77d","Type":"ContainerStarted","Data":"b90b528f7e7dbfbc0c62b11d52b5df649bd8b554c09f9a221243d4de5a3c5d81"} Jul 20 23:37:19.395503 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:19.395441 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" podStartSLOduration=1.3954272730000001 podStartE2EDuration="1.395427273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:37:19.394233637 +0000 UTC m=+2614.712585935" watchObservedRunningTime="2026-07-20 23:37:19.395427273 +0000 UTC m=+2614.713779559" Jul 20 23:37:23.341999 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.341953 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" podUID="6cd6e7db-b243-4b7a-a32a-be14abd81b5a" containerName="main" containerID="cri-o://022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253" gracePeriod=45 Jul 20 23:37:23.595512 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.595429 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:37:23.710734 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.710685 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-home\") pod \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " Jul 20 23:37:23.710734 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.710733 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-dshm\") pod \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " Jul 20 23:37:23.711092 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.710781 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzdt8\" (UniqueName: \"kubernetes.io/projected/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-kube-api-access-nzdt8\") pod \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " Jul 20 23:37:23.711092 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.710824 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tls-certs\") pod \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " Jul 20 23:37:23.711092 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.710855 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-model-cache\") pod \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " Jul 20 23:37:23.711092 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.710892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tmp-dir\") pod \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\" (UID: \"6cd6e7db-b243-4b7a-a32a-be14abd81b5a\") " Jul 20 23:37:23.711092 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.710989 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-home" (OuterVolumeSpecName: "home") pod "6cd6e7db-b243-4b7a-a32a-be14abd81b5a" (UID: "6cd6e7db-b243-4b7a-a32a-be14abd81b5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:37:23.711331 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.711159 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cd6e7db-b243-4b7a-a32a-be14abd81b5a" (UID: "6cd6e7db-b243-4b7a-a32a-be14abd81b5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:37:23.711331 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.711182 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cd6e7db-b243-4b7a-a32a-be14abd81b5a" (UID: "6cd6e7db-b243-4b7a-a32a-be14abd81b5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:37:23.711331 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.711224 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-home\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:37:23.712953 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.712923 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-dshm" (OuterVolumeSpecName: "dshm") pod "6cd6e7db-b243-4b7a-a32a-be14abd81b5a" (UID: "6cd6e7db-b243-4b7a-a32a-be14abd81b5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:37:23.712953 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.712944 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cd6e7db-b243-4b7a-a32a-be14abd81b5a" (UID: "6cd6e7db-b243-4b7a-a32a-be14abd81b5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:37:23.713130 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.713012 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-kube-api-access-nzdt8" (OuterVolumeSpecName: "kube-api-access-nzdt8") pod "6cd6e7db-b243-4b7a-a32a-be14abd81b5a" (UID: "6cd6e7db-b243-4b7a-a32a-be14abd81b5a"). InnerVolumeSpecName "kube-api-access-nzdt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:37:23.812543 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.812504 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-dshm\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:37:23.812543 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.812534 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzdt8\" (UniqueName: \"kubernetes.io/projected/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-kube-api-access-nzdt8\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:37:23.812543 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.812544 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tls-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:37:23.812543 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.812553 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-model-cache\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:37:23.812805 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:23.812561 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd6e7db-b243-4b7a-a32a-be14abd81b5a-tmp-dir\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:37:24.402694 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:24.402653 2574 generic.go:356] "Generic (PLEG): container finished" podID="6cd6e7db-b243-4b7a-a32a-be14abd81b5a" containerID="022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253" exitCode=0 Jul 20 23:37:24.403116 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:24.402714 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" Jul 20 23:37:24.403116 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:24.402733 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" event={"ID":"6cd6e7db-b243-4b7a-a32a-be14abd81b5a","Type":"ContainerDied","Data":"022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253"} Jul 20 23:37:24.403116 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:24.402778 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26" event={"ID":"6cd6e7db-b243-4b7a-a32a-be14abd81b5a","Type":"ContainerDied","Data":"530614565d93b82f1273b29aa191b1ac045ea6f50129c3cfbdf326f64584b105"} Jul 20 23:37:24.403116 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:24.402795 2574 scope.go:117] "RemoveContainer" containerID="022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253" Jul 20 23:37:24.411346 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:24.411323 2574 scope.go:117] "RemoveContainer" containerID="022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253" Jul 20 23:37:24.411679 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:37:24.411659 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253\": container with ID starting with 022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253 not found: ID does not exist" containerID="022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253" Jul 20 23:37:24.411733 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:24.411689 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253"} err="failed to get container status \"022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253\": rpc error: code = NotFound desc = could not find container \"022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253\": container with ID starting with 022c22cabbc71cad043d7fd063ab75abb90e2bc706feb39383ee7ec2dcbfd253 not found: ID does not exist" Jul 20 23:37:24.424167 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:24.424144 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26"] Jul 20 23:37:24.427807 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:24.427787 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/sub-test-model-107ae7-kserve-858f67c665-cpt26"] Jul 20 23:37:25.230354 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:25.230312 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd6e7db-b243-4b7a-a32a-be14abd81b5a" path="/var/lib/kubelet/pods/6cd6e7db-b243-4b7a-a32a-be14abd81b5a/volumes" Jul 20 23:37:28.511438 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:28.511403 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:28.511438 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:28.511447 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:28.523814 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:28.523788 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:37:29.439937 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:37:29.439899 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:38:17.832208 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:17.832166 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd"] Jul 20 23:38:32.852652 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:32.852585 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" podUID="c46cefdb-a599-4ed2-8985-37e63865f77d" containerName="main" containerID="cri-o://37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631" gracePeriod=45 Jul 20 23:38:33.106907 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.106844 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:38:33.264137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.264101 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-tmp-dir\") pod \"c46cefdb-a599-4ed2-8985-37e63865f77d\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " Jul 20 23:38:33.264335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.264157 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-model-cache\") pod \"c46cefdb-a599-4ed2-8985-37e63865f77d\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " Jul 20 23:38:33.264335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.264211 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-home\") pod \"c46cefdb-a599-4ed2-8985-37e63865f77d\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " Jul 20 23:38:33.264335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.264246 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46cefdb-a599-4ed2-8985-37e63865f77d-tls-certs\") pod \"c46cefdb-a599-4ed2-8985-37e63865f77d\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " Jul 20 23:38:33.264335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.264263 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-dshm\") pod \"c46cefdb-a599-4ed2-8985-37e63865f77d\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " Jul 20 23:38:33.264335 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.264289 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5qcm\" (UniqueName: \"kubernetes.io/projected/c46cefdb-a599-4ed2-8985-37e63865f77d-kube-api-access-r5qcm\") pod \"c46cefdb-a599-4ed2-8985-37e63865f77d\" (UID: \"c46cefdb-a599-4ed2-8985-37e63865f77d\") " Jul 20 23:38:33.264624 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.264492 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c46cefdb-a599-4ed2-8985-37e63865f77d" (UID: "c46cefdb-a599-4ed2-8985-37e63865f77d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:38:33.264624 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.264529 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-home" (OuterVolumeSpecName: "home") pod "c46cefdb-a599-4ed2-8985-37e63865f77d" (UID: "c46cefdb-a599-4ed2-8985-37e63865f77d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:38:33.264624 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.264540 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c46cefdb-a599-4ed2-8985-37e63865f77d" (UID: "c46cefdb-a599-4ed2-8985-37e63865f77d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:38:33.266535 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.266508 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-dshm" (OuterVolumeSpecName: "dshm") pod "c46cefdb-a599-4ed2-8985-37e63865f77d" (UID: "c46cefdb-a599-4ed2-8985-37e63865f77d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:38:33.266535 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.266520 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46cefdb-a599-4ed2-8985-37e63865f77d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c46cefdb-a599-4ed2-8985-37e63865f77d" (UID: "c46cefdb-a599-4ed2-8985-37e63865f77d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:38:33.266668 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.266535 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46cefdb-a599-4ed2-8985-37e63865f77d-kube-api-access-r5qcm" (OuterVolumeSpecName: "kube-api-access-r5qcm") pod "c46cefdb-a599-4ed2-8985-37e63865f77d" (UID: "c46cefdb-a599-4ed2-8985-37e63865f77d"). InnerVolumeSpecName "kube-api-access-r5qcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:38:33.366319 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.366219 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46cefdb-a599-4ed2-8985-37e63865f77d-tls-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:38:33.366319 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.366258 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-dshm\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:38:33.366319 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.366272 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qcm\" (UniqueName: \"kubernetes.io/projected/c46cefdb-a599-4ed2-8985-37e63865f77d-kube-api-access-r5qcm\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:38:33.366319 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.366287 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-tmp-dir\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:38:33.366319 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.366303 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-model-cache\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:38:33.366319 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.366316 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46cefdb-a599-4ed2-8985-37e63865f77d-home\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:38:33.743060 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.742973 2574 generic.go:356] "Generic (PLEG): container finished" podID="c46cefdb-a599-4ed2-8985-37e63865f77d" containerID="37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631" exitCode=0 Jul 20 23:38:33.743060 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.743017 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" Jul 20 23:38:33.743302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.743055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" event={"ID":"c46cefdb-a599-4ed2-8985-37e63865f77d","Type":"ContainerDied","Data":"37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631"} Jul 20 23:38:33.743302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.743104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd" event={"ID":"c46cefdb-a599-4ed2-8985-37e63865f77d","Type":"ContainerDied","Data":"b90b528f7e7dbfbc0c62b11d52b5df649bd8b554c09f9a221243d4de5a3c5d81"} Jul 20 23:38:33.743302 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.743126 2574 scope.go:117] "RemoveContainer" containerID="37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631" Jul 20 23:38:33.754420 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.754399 2574 scope.go:117] "RemoveContainer" containerID="37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631" Jul 20 23:38:33.754724 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:38:33.754703 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631\": container with ID starting with 37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631 not found: ID does not exist" containerID="37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631" Jul 20 23:38:33.754784 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.754733 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631"} err="failed to get container status \"37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631\": rpc error: code = NotFound desc = could not find container \"37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631\": container with ID starting with 37d7f0b33a2f95eeed20f05b26917a0828865177f8745aed0dfb94fe39f6f631 not found: ID does not exist" Jul 20 23:38:33.767686 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.767656 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd"] Jul 20 23:38:33.770995 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:33.770970 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-107ae7/rate-test-model-107ae7-kserve-5c4b987fd7-wcgnd"] Jul 20 23:38:35.232495 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:35.232438 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c46cefdb-a599-4ed2-8985-37e63865f77d" path="/var/lib/kubelet/pods/c46cefdb-a599-4ed2-8985-37e63865f77d/volumes" Jul 20 23:38:45.259109 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:45.259026 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:38:45.266848 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:45.266826 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sjc24_466d216d-acb1-4f99-9f98-819cc18b3367/ovn-acl-logging/0.log" Jul 20 23:38:47.925040 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:47.925003 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4"] Jul 20 23:38:47.925491 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:47.925256 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" podUID="86003d1f-0a06-4d68-b383-d280f75d8cc2" containerName="maas-api" containerID="cri-o://fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8" gracePeriod=30 Jul 20 23:38:48.180430 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.180373 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:38:48.190140 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.190116 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86003d1f-0a06-4d68-b383-d280f75d8cc2-openshift-service-ca\") pod \"86003d1f-0a06-4d68-b383-d280f75d8cc2\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " Jul 20 23:38:48.190251 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.190150 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86003d1f-0a06-4d68-b383-d280f75d8cc2-maas-api-tls\") pod \"86003d1f-0a06-4d68-b383-d280f75d8cc2\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " Jul 20 23:38:48.190251 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.190202 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vdz8\" (UniqueName: \"kubernetes.io/projected/86003d1f-0a06-4d68-b383-d280f75d8cc2-kube-api-access-7vdz8\") pod \"86003d1f-0a06-4d68-b383-d280f75d8cc2\" (UID: \"86003d1f-0a06-4d68-b383-d280f75d8cc2\") " Jul 20 23:38:48.190484 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.190439 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86003d1f-0a06-4d68-b383-d280f75d8cc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "86003d1f-0a06-4d68-b383-d280f75d8cc2" (UID: "86003d1f-0a06-4d68-b383-d280f75d8cc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:38:48.192307 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.192278 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86003d1f-0a06-4d68-b383-d280f75d8cc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "86003d1f-0a06-4d68-b383-d280f75d8cc2" (UID: "86003d1f-0a06-4d68-b383-d280f75d8cc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:38:48.192373 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.192335 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86003d1f-0a06-4d68-b383-d280f75d8cc2-kube-api-access-7vdz8" (OuterVolumeSpecName: "kube-api-access-7vdz8") pod "86003d1f-0a06-4d68-b383-d280f75d8cc2" (UID: "86003d1f-0a06-4d68-b383-d280f75d8cc2"). InnerVolumeSpecName "kube-api-access-7vdz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:38:48.291410 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.291373 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86003d1f-0a06-4d68-b383-d280f75d8cc2-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:38:48.291410 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.291405 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86003d1f-0a06-4d68-b383-d280f75d8cc2-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:38:48.291410 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.291414 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vdz8\" (UniqueName: \"kubernetes.io/projected/86003d1f-0a06-4d68-b383-d280f75d8cc2-kube-api-access-7vdz8\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:38:48.811964 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.811931 2574 generic.go:356] "Generic (PLEG): container finished" podID="86003d1f-0a06-4d68-b383-d280f75d8cc2" containerID="fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8" exitCode=0 Jul 20 23:38:48.812137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.811975 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" Jul 20 23:38:48.812137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.812017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" event={"ID":"86003d1f-0a06-4d68-b383-d280f75d8cc2","Type":"ContainerDied","Data":"fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8"} Jul 20 23:38:48.812137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.812068 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4" event={"ID":"86003d1f-0a06-4d68-b383-d280f75d8cc2","Type":"ContainerDied","Data":"1663ae0fdb93858300e5e5d8212fe308c4f59b3e85eb31e868477e45ca362f85"} Jul 20 23:38:48.812137 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.812089 2574 scope.go:117] "RemoveContainer" containerID="fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8" Jul 20 23:38:48.821221 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.821201 2574 scope.go:117] "RemoveContainer" containerID="fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8" Jul 20 23:38:48.821493 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:38:48.821452 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8\": container with ID starting with fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8 not found: ID does not exist" containerID="fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8" Jul 20 23:38:48.821547 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.821503 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8"} err="failed to get container status \"fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8\": rpc error: code = NotFound desc = could not find container \"fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8\": container with ID starting with fbc3955774c422ac32cc7716e09c8fc4bf5a76a145c4743c4b64828288a93bd8 not found: ID does not exist" Jul 20 23:38:48.833524 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.833492 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4"] Jul 20 23:38:48.835710 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:48.835691 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-107ae7-6dc8cc5fd5-7bmb4"] Jul 20 23:38:49.227660 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:49.227629 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86003d1f-0a06-4d68-b383-d280f75d8cc2" path="/var/lib/kubelet/pods/86003d1f-0a06-4d68-b383-d280f75d8cc2/volumes" Jul 20 23:38:58.931548 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:58.931506 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5"] Jul 20 23:38:58.932149 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:38:58.931750 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" containerID="cri-o://26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e" gracePeriod=30 Jul 20 23:39:01.014333 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:01.014287 2574 patch_prober.go:28] interesting pod/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 23:39:01.014720 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:01.014349 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 23:39:04.182303 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.182278 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:39:04.228005 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.227910 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-envoy\") pod \"0917a360-a964-479b-b304-3b49069bcdaf\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " Jul 20 23:39:04.228005 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.227966 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-data\") pod \"0917a360-a964-479b-b304-3b49069bcdaf\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " Jul 20 23:39:04.228243 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228007 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-credential-socket\") pod \"0917a360-a964-479b-b304-3b49069bcdaf\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " Jul 20 23:39:04.228243 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228045 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-socket\") pod \"0917a360-a964-479b-b304-3b49069bcdaf\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " Jul 20 23:39:04.228243 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228066 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0917a360-a964-479b-b304-3b49069bcdaf-istio-podinfo\") pod \"0917a360-a964-479b-b304-3b49069bcdaf\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " Jul 20 23:39:04.228243 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228114 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b99k\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-kube-api-access-4b99k\") pod \"0917a360-a964-479b-b304-3b49069bcdaf\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " Jul 20 23:39:04.228243 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228134 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-certs\") pod \"0917a360-a964-479b-b304-3b49069bcdaf\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " Jul 20 23:39:04.228243 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228206 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0917a360-a964-479b-b304-3b49069bcdaf-istiod-ca-cert\") pod \"0917a360-a964-479b-b304-3b49069bcdaf\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " Jul 20 23:39:04.228567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228257 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-istio-token\") pod \"0917a360-a964-479b-b304-3b49069bcdaf\" (UID: \"0917a360-a964-479b-b304-3b49069bcdaf\") " Jul 20 23:39:04.228567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228365 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0917a360-a964-479b-b304-3b49069bcdaf" (UID: "0917a360-a964-479b-b304-3b49069bcdaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:39:04.228567 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228379 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0917a360-a964-479b-b304-3b49069bcdaf" (UID: "0917a360-a964-479b-b304-3b49069bcdaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:39:04.228742 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228595 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-credential-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:04.228742 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228622 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-socket\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:04.228742 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228596 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0917a360-a964-479b-b304-3b49069bcdaf" (UID: "0917a360-a964-479b-b304-3b49069bcdaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:39:04.228917 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.228895 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "0917a360-a964-479b-b304-3b49069bcdaf" (UID: "0917a360-a964-479b-b304-3b49069bcdaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:39:04.229104 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.229077 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0917a360-a964-479b-b304-3b49069bcdaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0917a360-a964-479b-b304-3b49069bcdaf" (UID: "0917a360-a964-479b-b304-3b49069bcdaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:39:04.230328 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.230302 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0917a360-a964-479b-b304-3b49069bcdaf" (UID: "0917a360-a964-479b-b304-3b49069bcdaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:39:04.230643 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.230623 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0917a360-a964-479b-b304-3b49069bcdaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0917a360-a964-479b-b304-3b49069bcdaf" (UID: "0917a360-a964-479b-b304-3b49069bcdaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 23:39:04.230813 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.230797 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "0917a360-a964-479b-b304-3b49069bcdaf" (UID: "0917a360-a964-479b-b304-3b49069bcdaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:39:04.230944 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.230927 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-kube-api-access-4b99k" (OuterVolumeSpecName: "kube-api-access-4b99k") pod "0917a360-a964-479b-b304-3b49069bcdaf" (UID: "0917a360-a964-479b-b304-3b49069bcdaf"). InnerVolumeSpecName "kube-api-access-4b99k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:39:04.329542 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.329499 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0917a360-a964-479b-b304-3b49069bcdaf-istio-podinfo\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:04.329542 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.329537 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b99k\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-kube-api-access-4b99k\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:04.329542 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.329554 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-workload-certs\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:04.329788 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.329567 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0917a360-a964-479b-b304-3b49069bcdaf-istiod-ca-cert\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:04.329788 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.329580 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0917a360-a964-479b-b304-3b49069bcdaf-istio-token\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:04.329788 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.329591 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-envoy\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:04.329788 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.329603 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0917a360-a964-479b-b304-3b49069bcdaf-istio-data\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:04.890607 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.890568 2574 generic.go:356] "Generic (PLEG): container finished" podID="0917a360-a964-479b-b304-3b49069bcdaf" containerID="26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e" exitCode=0 Jul 20 23:39:04.890790 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.890636 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" Jul 20 23:39:04.890790 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.890643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" event={"ID":"0917a360-a964-479b-b304-3b49069bcdaf","Type":"ContainerDied","Data":"26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e"} Jul 20 23:39:04.890790 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.890724 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5" event={"ID":"0917a360-a964-479b-b304-3b49069bcdaf","Type":"ContainerDied","Data":"5c40e58ee639aa0379620c6d6e153e190826cd46d184dd43b7216b2f761f6777"} Jul 20 23:39:04.890790 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.890754 2574 scope.go:117] "RemoveContainer" containerID="26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e" Jul 20 23:39:04.899514 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.899495 2574 scope.go:117] "RemoveContainer" containerID="546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646" Jul 20 23:39:04.907538 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.907452 2574 scope.go:117] "RemoveContainer" containerID="26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e" Jul 20 23:39:04.907858 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:39:04.907838 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e\": container with ID starting with 26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e not found: ID does not exist" containerID="26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e" Jul 20 23:39:04.907904 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.907867 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e"} err="failed to get container status \"26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e\": rpc error: code = NotFound desc = could not find container \"26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e\": container with ID starting with 26b9639df188d9df1daa5e3e296ff3136529e9c29c22ac5dc895e4fa2763226e not found: ID does not exist" Jul 20 23:39:04.907904 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.907887 2574 scope.go:117] "RemoveContainer" containerID="546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646" Jul 20 23:39:04.908154 ip-10-0-129-209 kubenswrapper[2574]: E0720 23:39:04.908132 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646\": container with ID starting with 546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646 not found: ID does not exist" containerID="546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646" Jul 20 23:39:04.908209 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.908159 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646"} err="failed to get container status \"546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646\": rpc error: code = NotFound desc = could not find container \"546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646\": container with ID starting with 546d791bb65858c381a08f4289b556b7d16e3aade0e3a66b02d866c2ffe58646 not found: ID does not exist" Jul 20 23:39:04.912066 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.912038 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5"] Jul 20 23:39:04.916949 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:04.916920 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-107ae7-openshift-default-67499b8555-cdhd5"] Jul 20 23:39:05.229005 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:05.228930 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0917a360-a964-479b-b304-3b49069bcdaf" path="/var/lib/kubelet/pods/0917a360-a964-479b-b304-3b49069bcdaf/volumes" Jul 20 23:39:09.772357 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:09.772325 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4"] Jul 20 23:39:09.772870 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:09.772557 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" podUID="7bbd4768-4fa8-437a-8db8-719d27294423" containerName="maas-api" containerID="cri-o://27dd844f911415c45c7b54d8ab1679a4e3b59f893516dabcc29303e103c28a5e" gracePeriod=30 Jul 20 23:39:09.916636 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:09.916607 2574 generic.go:356] "Generic (PLEG): container finished" podID="7bbd4768-4fa8-437a-8db8-719d27294423" containerID="27dd844f911415c45c7b54d8ab1679a4e3b59f893516dabcc29303e103c28a5e" exitCode=0 Jul 20 23:39:09.916803 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:09.916681 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" event={"ID":"7bbd4768-4fa8-437a-8db8-719d27294423","Type":"ContainerDied","Data":"27dd844f911415c45c7b54d8ab1679a4e3b59f893516dabcc29303e103c28a5e"} Jul 20 23:39:10.020860 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.020835 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:39:10.077500 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.077376 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hgzg\" (UniqueName: \"kubernetes.io/projected/7bbd4768-4fa8-437a-8db8-719d27294423-kube-api-access-4hgzg\") pod \"7bbd4768-4fa8-437a-8db8-719d27294423\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " Jul 20 23:39:10.077500 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.077428 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4768-4fa8-437a-8db8-719d27294423-openshift-service-ca\") pod \"7bbd4768-4fa8-437a-8db8-719d27294423\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " Jul 20 23:39:10.077766 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.077548 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd4768-4fa8-437a-8db8-719d27294423-maas-api-tls\") pod \"7bbd4768-4fa8-437a-8db8-719d27294423\" (UID: \"7bbd4768-4fa8-437a-8db8-719d27294423\") " Jul 20 23:39:10.077895 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.077869 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbd4768-4fa8-437a-8db8-719d27294423-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bbd4768-4fa8-437a-8db8-719d27294423" (UID: "7bbd4768-4fa8-437a-8db8-719d27294423"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:39:10.079570 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.079536 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbd4768-4fa8-437a-8db8-719d27294423-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bbd4768-4fa8-437a-8db8-719d27294423" (UID: "7bbd4768-4fa8-437a-8db8-719d27294423"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:39:10.079679 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.079583 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbd4768-4fa8-437a-8db8-719d27294423-kube-api-access-4hgzg" (OuterVolumeSpecName: "kube-api-access-4hgzg") pod "7bbd4768-4fa8-437a-8db8-719d27294423" (UID: "7bbd4768-4fa8-437a-8db8-719d27294423"). InnerVolumeSpecName "kube-api-access-4hgzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:39:10.178422 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.178387 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd4768-4fa8-437a-8db8-719d27294423-maas-api-tls\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:10.178422 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.178416 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hgzg\" (UniqueName: \"kubernetes.io/projected/7bbd4768-4fa8-437a-8db8-719d27294423-kube-api-access-4hgzg\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:10.178422 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.178426 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd4768-4fa8-437a-8db8-719d27294423-openshift-service-ca\") on node \"ip-10-0-129-209.ec2.internal\" DevicePath \"\"" Jul 20 23:39:10.922836 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.922812 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" Jul 20 23:39:10.923234 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.922808 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4" event={"ID":"7bbd4768-4fa8-437a-8db8-719d27294423","Type":"ContainerDied","Data":"20c72a071a8f8bffb4ce8f032d6972e520ed03e9120527c195a2fff623ad4863"} Jul 20 23:39:10.923234 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.922986 2574 scope.go:117] "RemoveContainer" containerID="27dd844f911415c45c7b54d8ab1679a4e3b59f893516dabcc29303e103c28a5e" Jul 20 23:39:10.944662 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.944633 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4"] Jul 20 23:39:10.948532 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:10.948507 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b9afe-699d5b589f-hkhk4"] Jul 20 23:39:11.227614 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:11.227533 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bbd4768-4fa8-437a-8db8-719d27294423" path="/var/lib/kubelet/pods/7bbd4768-4fa8-437a-8db8-719d27294423/volumes" Jul 20 23:39:21.663071 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:21.663038 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-769d7847f9-kbhrz_6c141d47-8a38-428d-bb1d-e77acd66cb92/authorino/0.log" Jul 20 23:39:26.832689 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:26.832662 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7t68m_cf87b6dd-57f0-4a91-8750-f7599613751d/manager/0.log" Jul 20 23:39:26.945841 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:26.945809 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bn26h_d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2/manager/0.log" Jul 20 23:39:27.061648 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:27.061614 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4874f67f-pckkq_f3d819a3-29f2-4992-b070-c85423fa6d1e/manager/0.log" Jul 20 23:39:27.176209 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:27.176183 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tnwnd_d9775d2c-3396-4de8-8f9a-58fb144581d0/server/0.log" Jul 20 23:39:27.534266 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:27.534184 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fd79659-kxtvp_b91ee607-5707-4410-9e32-ee61b9ea3204/copy-manifests/0.log" Jul 20 23:39:27.540705 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:27.540679 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fd79659-kxtvp_b91ee607-5707-4410-9e32-ee61b9ea3204/manager/0.log" Jul 20 23:39:28.001956 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:28.001928 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5f68cf4fbd-pj6bz_d276f814-e5c0-4612-9104-a316a8c870d6/maas-api/0.log" Jul 20 23:39:28.325912 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:28.325827 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-r257d_d72ea9c8-6b7e-402f-aee8-bf20537bc4fc/postgres/0.log" Jul 20 23:39:29.561955 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:29.561926 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-769d7847f9-kbhrz_6c141d47-8a38-428d-bb1d-e77acd66cb92/authorino/0.log" Jul 20 23:39:29.673348 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:29.673322 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tvhxj_eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a/manager/0.log" Jul 20 23:39:29.892737 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:29.892707 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-mqrvx_17011a37-8053-49ef-a6a2-23cee3a83dd9/kuadrant-console-plugin/0.log" Jul 20 23:39:30.009716 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:30.009687 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-9kkp2_9555a46d-3328-4463-b43a-36a4ccafd374/registry-server/0.log" Jul 20 23:39:30.348480 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:30.348446 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x92zq_b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9/manager/0.log" Jul 20 23:39:30.907307 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:30.907275 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d6bc8fc7-6m8t5_8dc6d214-d9c2-4e4e-9b75-51a41b9c303c/kube-auth-proxy/0.log" Jul 20 23:39:31.129683 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:31.129651 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-49zl4_5bf0978f-511a-4503-a7c0-8321d105b86f/istio-proxy/1.log" Jul 20 23:39:31.238488 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:31.238399 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-mt455_5fc0e005-0b90-4d48-9213-9d8ece94a5b7/payload-pre-processing/0.log" Jul 20 23:39:31.462997 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:31.462971 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86f6bc7688-gdrps_1395ccea-0612-4422-8bcc-050d534f6697/router/0.log" Jul 20 23:39:31.795759 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:31.795730 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-j5zqj_cc38ab44-9d53-4c4c-bc95-f2d19a0818eb/main/0.log" Jul 20 23:39:31.903857 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:31.903828 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-zq5gx_1001a1c9-445c-4e3b-9605-ae5f3d952f35/main/0.log" Jul 20 23:39:41.033519 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:41.033491 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-574vd_5be5526a-0c29-42b3-91fb-6c6863746390/global-pull-secret-syncer/0.log" Jul 20 23:39:41.198881 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:41.198854 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kxxtg_36201deb-a807-451a-9a83-7446f325f7bc/konnectivity-agent/0.log" Jul 20 23:39:41.256909 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:41.256883 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-209.ec2.internal_4f2e386cf084133e9c2b8d9bc35bd306/haproxy/0.log" Jul 20 23:39:46.762174 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:46.762090 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-769d7847f9-kbhrz_6c141d47-8a38-428d-bb1d-e77acd66cb92/authorino/0.log" Jul 20 23:39:46.799889 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:46.799855 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tvhxj_eb043f2e-50ab-49d9-bfe6-1a6c960ebe6a/manager/0.log" Jul 20 23:39:46.866946 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:46.866917 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-mqrvx_17011a37-8053-49ef-a6a2-23cee3a83dd9/kuadrant-console-plugin/0.log" Jul 20 23:39:46.909811 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:46.909779 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-9kkp2_9555a46d-3328-4463-b43a-36a4ccafd374/registry-server/0.log" Jul 20 23:39:47.149778 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:47.149749 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x92zq_b318e2e7-65e8-4a2e-8b33-a1a2cdee35b9/manager/0.log" Jul 20 23:39:48.982004 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:48.981969 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-6d87c_47303d5f-832f-48f6-98ea-3aa738f34b38/cluster-monitoring-operator/0.log" Jul 20 23:39:49.290779 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:49.290706 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zshqq_f87265f1-e5ae-4d8e-ad1a-b82996c5578f/node-exporter/0.log" Jul 20 23:39:49.315878 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:49.315853 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zshqq_f87265f1-e5ae-4d8e-ad1a-b82996c5578f/kube-rbac-proxy/0.log" Jul 20 23:39:49.340593 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:49.340568 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zshqq_f87265f1-e5ae-4d8e-ad1a-b82996c5578f/init-textfile/0.log" Jul 20 23:39:51.052305 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052273 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg"] Jul 20 23:39:51.052697 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052633 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" Jul 20 23:39:51.052697 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052644 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" Jul 20 23:39:51.052697 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052654 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86003d1f-0a06-4d68-b383-d280f75d8cc2" containerName="maas-api" Jul 20 23:39:51.052697 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052659 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="86003d1f-0a06-4d68-b383-d280f75d8cc2" containerName="maas-api" Jul 20 23:39:51.052697 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052668 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bbd4768-4fa8-437a-8db8-719d27294423" containerName="maas-api" Jul 20 23:39:51.052697 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052674 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bbd4768-4fa8-437a-8db8-719d27294423" containerName="maas-api" Jul 20 23:39:51.052697 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052691 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" Jul 20 23:39:51.052697 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052696 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052703 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd6e7db-b243-4b7a-a32a-be14abd81b5a" containerName="main" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052708 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd6e7db-b243-4b7a-a32a-be14abd81b5a" containerName="main" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052714 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c46cefdb-a599-4ed2-8985-37e63865f77d" containerName="main" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052719 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c46cefdb-a599-4ed2-8985-37e63865f77d" containerName="main" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052765 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052773 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="86003d1f-0a06-4d68-b383-d280f75d8cc2" containerName="maas-api" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052778 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c46cefdb-a599-4ed2-8985-37e63865f77d" containerName="main" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052783 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0917a360-a964-479b-b304-3b49069bcdaf" containerName="istio-proxy" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052789 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd6e7db-b243-4b7a-a32a-be14abd81b5a" containerName="main" Jul 20 23:39:51.052948 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.052794 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bbd4768-4fa8-437a-8db8-719d27294423" containerName="maas-api" Jul 20 23:39:51.057231 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.057209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.059959 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.059940 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95lz9\"/\"kube-root-ca.crt\"" Jul 20 23:39:51.060070 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.059941 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95lz9\"/\"openshift-service-ca.crt\"" Jul 20 23:39:51.060976 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.060961 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-95lz9\"/\"default-dockercfg-rh8pd\"" Jul 20 23:39:51.064648 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.064629 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg"] Jul 20 23:39:51.121146 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.121109 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-lib-modules\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.121146 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.121145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8crd\" (UniqueName: \"kubernetes.io/projected/cef68d3e-1898-4403-af34-0fc04ca718d0-kube-api-access-r8crd\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.121407 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.121185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-sys\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.121407 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.121241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-podres\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.121407 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.121280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-proc\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.221920 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.221881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-podres\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.222130 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.221948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-proc\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.222130 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.222004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-lib-modules\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.222130 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.222030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8crd\" (UniqueName: \"kubernetes.io/projected/cef68d3e-1898-4403-af34-0fc04ca718d0-kube-api-access-r8crd\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.222130 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.222067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-podres\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.222130 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.222073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-sys\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.222130 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.222071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-proc\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.222348 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.222139 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-lib-modules\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.222348 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.222124 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cef68d3e-1898-4403-af34-0fc04ca718d0-sys\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.230639 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.230614 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8crd\" (UniqueName: \"kubernetes.io/projected/cef68d3e-1898-4403-af34-0fc04ca718d0-kube-api-access-r8crd\") pod \"perf-node-gather-daemonset-dlpcg\" (UID: \"cef68d3e-1898-4403-af34-0fc04ca718d0\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.368183 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.368149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:51.501968 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:51.501942 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg"] Jul 20 23:39:51.503552 ip-10-0-129-209 kubenswrapper[2574]: W0720 23:39:51.503525 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcef68d3e_1898_4403_af34_0fc04ca718d0.slice/crio-fb9ea79df64de0c384b74df9725a30dd720b8eeeb1bfea5bc675c9d6fd81dfaf WatchSource:0}: Error finding container fb9ea79df64de0c384b74df9725a30dd720b8eeeb1bfea5bc675c9d6fd81dfaf: Status 404 returned error can't find the container with id fb9ea79df64de0c384b74df9725a30dd720b8eeeb1bfea5bc675c9d6fd81dfaf Jul 20 23:39:52.112209 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:52.112167 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" event={"ID":"cef68d3e-1898-4403-af34-0fc04ca718d0","Type":"ContainerStarted","Data":"dcd343f176f35117bf6f749dd7a2e91888dcb6aeb7e69b3fe99f429123768654"} Jul 20 23:39:52.112689 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:52.112230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" event={"ID":"cef68d3e-1898-4403-af34-0fc04ca718d0","Type":"ContainerStarted","Data":"fb9ea79df64de0c384b74df9725a30dd720b8eeeb1bfea5bc675c9d6fd81dfaf"} Jul 20 23:39:52.112689 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:52.112317 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:52.129195 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:52.129140 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" podStartSLOduration=1.129124742 podStartE2EDuration="1.129124742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 23:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:39:52.127889482 +0000 UTC m=+2767.446241770" watchObservedRunningTime="2026-07-20 23:39:52.129124742 +0000 UTC m=+2767.447477069" Jul 20 23:39:54.041909 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:54.041875 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mm8pc_e5af0766-d27e-4caf-bb82-e6929c31b8f9/dns/0.log" Jul 20 23:39:54.062731 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:54.062709 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mm8pc_e5af0766-d27e-4caf-bb82-e6929c31b8f9/kube-rbac-proxy/0.log" Jul 20 23:39:54.108845 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:54.108824 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jx6hc_e84fa919-76a7-476e-9657-c195f0217bbc/dns-node-resolver/0.log" Jul 20 23:39:54.662779 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:54.662748 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84df66f6d-lzwgx_359ef3fc-983f-4e37-982c-76b9ffb1adb5/registry/0.log" Jul 20 23:39:54.708902 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:54.708876 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cmrt4_0474b40b-5711-4a08-a644-f9caf4afed37/node-ca/0.log" Jul 20 23:39:55.778305 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:55.778277 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d6bc8fc7-6m8t5_8dc6d214-d9c2-4e4e-9b75-51a41b9c303c/kube-auth-proxy/0.log" Jul 20 23:39:55.837547 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:55.837511 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-49zl4_5bf0978f-511a-4503-a7c0-8321d105b86f/istio-proxy/0.log" Jul 20 23:39:55.867575 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:55.867547 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-49zl4_5bf0978f-511a-4503-a7c0-8321d105b86f/istio-proxy/1.log" Jul 20 23:39:55.899180 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:55.899139 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-mt455_5fc0e005-0b90-4d48-9213-9d8ece94a5b7/payload-pre-processing/0.log" Jul 20 23:39:55.949313 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:55.949289 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86f6bc7688-gdrps_1395ccea-0612-4422-8bcc-050d534f6697/router/0.log" Jul 20 23:39:56.685839 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:56.685812 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rhwct_7febacb1-bee8-4309-9751-6e2d693f2bcc/serve-healthcheck-canary/0.log" Jul 20 23:39:57.186492 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:57.186447 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-clnd8_07901246-5ff6-4ada-b81f-6b41ac1df630/kube-rbac-proxy/0.log" Jul 20 23:39:57.206942 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:57.206911 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-clnd8_07901246-5ff6-4ada-b81f-6b41ac1df630/exporter/0.log" Jul 20 23:39:57.228192 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:57.228164 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-clnd8_07901246-5ff6-4ada-b81f-6b41ac1df630/extractor/0.log" Jul 20 23:39:58.126303 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:58.126279 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-dlpcg" Jul 20 23:39:59.470992 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:59.470965 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7t68m_cf87b6dd-57f0-4a91-8750-f7599613751d/manager/0.log" Jul 20 23:39:59.642236 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:59.642205 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bn26h_d4474cbe-0c59-4d8e-8cc5-e90b3eda8ba2/manager/0.log" Jul 20 23:39:59.763801 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:59.763771 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4874f67f-pckkq_f3d819a3-29f2-4992-b070-c85423fa6d1e/manager/0.log" Jul 20 23:39:59.783449 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:59.783421 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tnwnd_d9775d2c-3396-4de8-8f9a-58fb144581d0/server/0.log" Jul 20 23:39:59.955703 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:59.955667 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fd79659-kxtvp_b91ee607-5707-4410-9e32-ee61b9ea3204/manager/0.log" Jul 20 23:39:59.983193 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:39:59.983117 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fd79659-kxtvp_b91ee607-5707-4410-9e32-ee61b9ea3204/copy-manifests/0.log" Jul 20 23:40:02.056585 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:02.056559 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d6bc8fc7-6m8t5_8dc6d214-d9c2-4e4e-9b75-51a41b9c303c/kube-auth-proxy/0.log" Jul 20 23:40:02.099495 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:02.099439 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-49zl4_5bf0978f-511a-4503-a7c0-8321d105b86f/istio-proxy/0.log" Jul 20 23:40:02.124861 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:02.124832 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-49zl4_5bf0978f-511a-4503-a7c0-8321d105b86f/istio-proxy/1.log" Jul 20 23:40:02.144587 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:02.144556 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-mt455_5fc0e005-0b90-4d48-9213-9d8ece94a5b7/payload-pre-processing/0.log" Jul 20 23:40:02.180386 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:02.180353 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86f6bc7688-gdrps_1395ccea-0612-4422-8bcc-050d534f6697/router/0.log" Jul 20 23:40:03.506810 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:03.506780 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d6bc8fc7-6m8t5_8dc6d214-d9c2-4e4e-9b75-51a41b9c303c/kube-auth-proxy/0.log" Jul 20 23:40:03.545764 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:03.545715 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-49zl4_5bf0978f-511a-4503-a7c0-8321d105b86f/istio-proxy/0.log" Jul 20 23:40:03.576000 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:03.575972 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-49zl4_5bf0978f-511a-4503-a7c0-8321d105b86f/istio-proxy/1.log" Jul 20 23:40:03.597725 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:03.597699 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-mt455_5fc0e005-0b90-4d48-9213-9d8ece94a5b7/payload-pre-processing/0.log" Jul 20 23:40:03.632569 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:03.632546 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86f6bc7688-gdrps_1395ccea-0612-4422-8bcc-050d534f6697/router/0.log" Jul 20 23:40:08.317008 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:08.316978 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8mn7_1bad4f19-6640-4c47-8340-af3bbaf3b736/kube-multus-additional-cni-plugins/0.log" Jul 20 23:40:08.339363 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:08.339334 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8mn7_1bad4f19-6640-4c47-8340-af3bbaf3b736/egress-router-binary-copy/0.log" Jul 20 23:40:08.360819 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:08.360793 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8mn7_1bad4f19-6640-4c47-8340-af3bbaf3b736/cni-plugins/0.log" Jul 20 23:40:08.382598 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:08.382574 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8mn7_1bad4f19-6640-4c47-8340-af3bbaf3b736/bond-cni-plugin/0.log" Jul 20 23:40:08.408348 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:08.408325 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8mn7_1bad4f19-6640-4c47-8340-af3bbaf3b736/routeoverride-cni/0.log" Jul 20 23:40:08.430977 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:08.430949 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8mn7_1bad4f19-6640-4c47-8340-af3bbaf3b736/whereabouts-cni-bincopy/0.log" Jul 20 23:40:08.452584 ip-10-0-129-209 kubenswrapper[2574]: I0720 23:40:08.452553 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8mn7_1bad4f19-6640-4c47-8340-af3bbaf3b736/whereabouts-cni/0.log"