Sep 16 18:39:32.066177 ip-10-0-138-234 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:39:32.437012 ip-10-0-138-234 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:32.437012 ip-10-0-138-234 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:39:32.437012 ip-10-0-138-234 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:32.437012 ip-10-0-138-234 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:39:32.437012 ip-10-0-138-234 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:32.437859 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.437779    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:39:32.443182 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443167    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:32.443182 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443182    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443186    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443190    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443194    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443197    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443199    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443202    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443205    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443208    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443211    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443214    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443216    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443219    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443222    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443225    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443228    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443230    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443232    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443235    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:32.443237 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443238    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443248    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443253    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443257    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443260    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443263    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443265    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443268    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443271    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443273    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443275    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443278    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443280    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443283    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443286    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443290    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443294    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443296    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443299    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:32.443675 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443302    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443304    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443306    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443310    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443312    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443315    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443317    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443319    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443322    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443324    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443329    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443332    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443335    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443338    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443341    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443345    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443348    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443351    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443354    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:32.444198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443356    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443359    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443361    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443364    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443366    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443369    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443372    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443375    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443377    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443380    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443382    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443385    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443387    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443390    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443393    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443395    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443398    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443401    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443403    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443406    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:32.445006 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443408    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443411    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443413    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443416    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443418    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443420    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443423    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443425    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443429    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443432    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443434    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443437    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443440    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443443    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443446    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443448    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443451    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443453    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443456    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:32.445681 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443459    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:32.446161 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.443461    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:32.447998 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.447981    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:32.447998 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.447997    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448001    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448004    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448008    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448012    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448015    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448018    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448021    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448024    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448028    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448030    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448033    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448036    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448039    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448041    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448043    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448046    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448049    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448051    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448054    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:32.448063 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448056    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448059    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448061    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448065    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448068    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448070    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448073    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448077    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448081    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448085    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448087    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448090    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448093    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448096    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448101    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448106    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448109    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448112    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448115    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:32.448551 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448118    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448120    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448123    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448125    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448128    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448130    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448133    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448135    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448138    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448140    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448144    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448146    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448149    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448152    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448154    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448157    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448160    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448163    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448166    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448168    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:32.449056 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448171    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448173    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448176    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448178    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448181    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448183    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448186    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448189    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448192    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448194    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448197    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448199    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448202    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448204    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448207    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448209    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448212    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448215    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448219    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448222    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:32.449541 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448224    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448227    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448229    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448233    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448237    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448241    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448243    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448246    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448249    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448252    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448254    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448257    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448260    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448262    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448265    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448268    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448270    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448273    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448275    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:32.450036 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448658    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448663    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448666    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448668    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448671    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448674    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448677    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448679    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448682    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448684    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448687    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448690    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448693    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448695    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448698    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448700    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448704    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448724    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448729    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:32.450510 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448734    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448738    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448741    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448744    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448747    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448750    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448753    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448756    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448759    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448762    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448765    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448767    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448770    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448772    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448775    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448778    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448780    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448783    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448785    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448788    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:32.451002 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448790    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448793    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448796    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448799    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448802    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448805    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448808    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448810    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448813    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448816    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448820    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448822    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448825    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448827    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448830    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448832    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448835    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448837    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448840    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448843    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:32.451482 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448845    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448848    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448850    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448853    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448855    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448858    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448860    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448862    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448865    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448867    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448870    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448873    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448875    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448879    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448882    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448885    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448888    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448891    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448893    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448896    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:32.451986 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448899    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448902    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448904    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448907    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448910    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448912    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448915    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448917    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448920    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448924    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448928    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448931    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448934    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448936    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448939    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448941    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448944    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448946    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448949    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:32.452535 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.448951    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449024    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449061    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449068    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449073    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449077    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449080    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449084    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449088    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449092    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449095    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449098    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449101    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449104    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449108    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449111    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449114    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449118    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449121    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449127    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449130    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449133    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449136    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449139    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:39:32.453008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449143    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449146    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449150    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449153    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449156    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449159    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449162    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449165    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449168    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449175    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449178    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449181    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449184    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449187    2575 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449190    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449194    2575 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449197    2575 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449201    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449205    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449208    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449211    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449214    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449217    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449220    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449223    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:39:32.453585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449225    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449230    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449233    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449236    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449239    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449242    2575 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449246    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449249    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449252    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449256    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449259    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449262    2575 flags.go:64] FLAG: --help="false"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449265    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449268    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449275    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449278    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449281    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449285    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449288    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449291    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449293    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449296    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449299    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449302    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:39:32.454203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449305    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449308    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449311    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449314    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449316    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449319    2575 flags.go:64] FLAG: --lock-file=""
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449322    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449325    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449328    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449333    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449337    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449340    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449342    2575 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449346    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449349    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449352    2575 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449355    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449359    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449362    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449366    2575 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449369    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449372    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449377    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449380    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449383    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:39:32.454838 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449386    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449389    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449396    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449400    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449403    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449406    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449409    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449415    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449417    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449421    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449424    2575 flags.go:64] FLAG: --port="10250"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449427    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449430    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc9fc037bfd44df9"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449433    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449436    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449439    2575 flags.go:64] FLAG: --register-node="true"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449442    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449446    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449449    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449452    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449455    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449459    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449462    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449465    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449468    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:39:32.455540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449471    2575 flags.go:64] FLAG: --runonce="false"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449474    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449476    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449479    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449482    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449487    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449490    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449493    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449496    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449500    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449503    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449506    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449508    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449512    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449515    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449518    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449523    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449526    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449529    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449533    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449536    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449539    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449542    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449545    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449549    2575 flags.go:64] FLAG: --v="2"
Sep 16 18:39:32.456195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449552    2575 flags.go:64] FLAG: --version="false"
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449557    2575 flags.go:64] FLAG: --vmodule=""
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449561    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449564    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449677    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449680    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449683    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449685    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449688    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449691    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449694    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449697    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449701    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449703    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449721    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449725    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449728    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449730    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449733    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449736    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:32.456869 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449740    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449743    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449746    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449749    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449752    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449755    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449757    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449760    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449763    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449765    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449768    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449770    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449773    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449775    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449780    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449783    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449786    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449788    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449791    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449794    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:32.457345 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449796    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449799    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449801    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449804    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449808    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449810    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449813    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449815    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449818    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449821    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449823    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449826    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449830    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449834    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449836    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449839    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449842    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449845    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449848    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:32.457833 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449851    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449854    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449856    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449858    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449861    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449863    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449866    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449871    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449874    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449876    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449879    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449881    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449884    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449886    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449889    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449891    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449894    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449898    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449901    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449904    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:32.458305 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449906    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449909    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449912    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449916    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449919    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449922    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449925    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449927    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449930    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449933    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449935    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449938    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449941    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449943    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449946    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449948    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449951    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449953    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449956    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:32.458810 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449960    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449962    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449965    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449968    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.449971    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.449980    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.456533    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.456548    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.456566    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.456571    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456624    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456629    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456632    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456635    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456638    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:32.459262 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456640    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456644    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456646    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456649    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456652    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456654    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456657    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456660    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456662    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456665    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456667    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456670    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456673    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456675    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456678    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456680    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456683    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456685    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456688    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456690    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:32.459646 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456693    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456695    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456698    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456700    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456705    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456722    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456726    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456729    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456732    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456734    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456737    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456740    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456742    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456745    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456747    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456750    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456753    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456756    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456758    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456761    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:32.460207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456763    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456766    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456769    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456771    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456774    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456776    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456779    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456781    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456784    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456788    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456792    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456794    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456797    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456799    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456803    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456807    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456810    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456814    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456817    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456820    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:32.460706 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456823    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456827    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456831    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456834    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456836    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456839    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456842    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456844    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456847    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456850    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456852    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456855    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456858    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456860    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456863    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456865    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456868    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456871    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456873    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:32.461188 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456876    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456879    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456881    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456884    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456886    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456889    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456891    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456894    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456897    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456900    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456903    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456906    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456908    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456911    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:32.461676 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.456913    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.456919    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457025    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457029    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457032    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457035    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457038    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457041    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457044    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457046    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457049    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457051    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457053    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457056    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457059    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:32.462033 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457061    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457064    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457066    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457069    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457071    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457074    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457076    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457078    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457081    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457083    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457086    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457088    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457091    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457094    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457097    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457100    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457102    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457105    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457107    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457110    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:32.462397 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457112    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457114    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457117    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457119    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457122    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457124    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457127    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457129    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457132    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457134    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457137    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457139    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457142    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457144    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457148    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457152    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457155    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457159    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457162    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:32.462928 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457165    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457168    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457170    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457172    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457175    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457178    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457181    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457183    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457186    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457189    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457191    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457194    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457196    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457199    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457201    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457204    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457208    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457211    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457214    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:32.463380 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457217    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457219    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457222    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457225    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457227    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457230    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457233    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457235    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457238    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457240    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457243    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457246    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457248    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457251    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457253    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457256    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457258    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457261    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457263    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457266    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:32.463850 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457269    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457271    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457274    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457280    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457283    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457286    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457288    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:32.457291    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.457296    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.457388    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.459336    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.460238    2575 server.go:1019] "Starting client certificate rotation"
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.460338    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:32.464321 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.460366    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:32.464650 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.462587    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:32.464650 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.464306    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:32.479248 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.479226    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:39:32.482188 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.482170    2575 log.go:25] "Validated CRI v1 image API"
Sep 16 18:39:32.482978 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.482962    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:39:32.485015 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.484991    2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 af067576-6c3a-4414-8b79-b4d2f4a9ca25:/dev/nvme0n1p4 e90ed447-c005-4024-80f7-6744d7f870e0:/dev/nvme0n1p3]
Sep 16 18:39:32.485089 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.485013    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:39:32.489841 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.489725    2575 manager.go:216] Machine: {Timestamp:2026-09-16 18:39:32.488623814 +0000 UTC m=+0.305389391 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093877 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f94f046b088a2e5df731cd80d489d SystemUUID:ec2f94f0-46b0-88a2-e5df-731cd80d489d BootID:b4e92aef-6a50-4692-9dd1-70ae0b37b436 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ef:2b:c4:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ef:2b:c4:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:30:4e:45:56:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:39:32.489841 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.489833    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:39:32.490000 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.489986    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:39:32.491572 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.491555    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:32.493004 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.492974    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:39:32.493171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.493008    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-234.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:39:32.493697 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.493685    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:39:32.493767 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.493704    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:39:32.493767 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.493730    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:32.494333 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.494321    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:32.495102 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.495090    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:39:32.497721 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.497693    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:32.497906 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.497894    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:39:32.497965 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.497936    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:39:32.497965 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.497951    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:39:32.498044 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.497966    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:39:32.498044 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.497980    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:39:32.498044 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.497992    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:39:32.498360 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.498346    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:32.498428 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.498370    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:32.500461 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.500447    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:39:32.502082 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.502069    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:39:32.502132 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.502099    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:39:32.503389 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503378    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:39:32.503445 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503393    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:39:32.503445 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503400    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:39:32.503445 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503405    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:39:32.503445 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503416    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:39:32.503445 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503442    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:39:32.503573 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503457    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:39:32.503573 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503462    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:39:32.503573 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503469    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:39:32.503573 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503474    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:39:32.503573 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503489    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:39:32.503573 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.503495    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:39:32.505291 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.505281    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:39:32.505291 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.505291    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:39:32.509181 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.509168    2575 server.go:1268] "Started kubelet"
Sep 16 18:39:32.509361 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.509296    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:39:32.509416 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.509360    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:39:32.509469 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.509436    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:39:32.509614 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.509586    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-234.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:39:32.509746 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.509699    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:39:32.509806 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.509792    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:39:32.510017 ip-10-0-138-234 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:39:32.515739 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.515652    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:39:32.516846 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.516826    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:39:32.517039 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.517015    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:39:32.518748 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.518728    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:32.519287 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.519265    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:39:32.519434 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.519391    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:32.519930 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.519781    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:39:32.519930 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.519798    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:39:32.519930 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.519890    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:39:32.520100 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.519968    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:39:32.520100 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.519983    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:39:32.520190 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.520159    2575 factory.go:55] Registering systemd factory
Sep 16 18:39:32.520190 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.520175    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:39:32.520430 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.520414    2575 factory.go:153] Registering CRI-O factory
Sep 16 18:39:32.520484 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.520430    2575 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:39:32.520484 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.520477    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:39:32.520588 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.520496    2575 factory.go:103] Registering Raw factory
Sep 16 18:39:32.520588 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.520513    2575 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:39:32.520887 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.519981    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-234.ec2.internal.18d5e1243123c06d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-234.ec2.internal,UID:ip-10-0-138-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-234.ec2.internal,},FirstTimestamp:2026-09-16 18:39:32.509139053 +0000 UTC m=+0.325904629,LastTimestamp:2026-09-16 18:39:32.509139053 +0000 UTC m=+0.325904629,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-234.ec2.internal,}"
Sep 16 18:39:32.520981 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.520918    2575 manager.go:349] Starting recovery of all containers
Sep 16 18:39:32.527461 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.527427    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:39:32.527631 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.527607    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:39:32.528590 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.528575    2575 manager.go:354] Recovery completed
Sep 16 18:39:32.534315 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.534300    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:32.535818 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.535791    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:32.535894 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.535830    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:32.535894 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.535844    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:32.537017 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.536999    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:39:32.537086 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.537018    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:39:32.537086 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.537040    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:32.537086 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.537003    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:32.537750 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.537733    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:32.537828 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.537765    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:32.537828 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.537776    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:32.537828 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.537756    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-234.ec2.internal.18d5e12432bace6a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-234.ec2.internal,UID:ip-10-0-138-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-234.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-234.ec2.internal,},FirstTimestamp:2026-09-16 18:39:32.535815786 +0000 UTC m=+0.352581364,LastTimestamp:2026-09-16 18:39:32.535815786 +0000 UTC m=+0.352581364,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-234.ec2.internal,}"
Sep 16 18:39:32.538234 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.538223    2575 policy_none.go:49] "None policy: Start"
Sep 16 18:39:32.538274 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.538240    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:39:32.538274 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.538253    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:39:32.539270 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.539259    2575 policy_none.go:47] "Start"
Sep 16 18:39:32.546903 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.546819    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-234.ec2.internal.18d5e12432bb1aa1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-234.ec2.internal,UID:ip-10-0-138-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-234.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-234.ec2.internal,},FirstTimestamp:2026-09-16 18:39:32.535835297 +0000 UTC m=+0.352600874,LastTimestamp:2026-09-16 18:39:32.535835297 +0000 UTC m=+0.352600874,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-234.ec2.internal,}"
Sep 16 18:39:32.549737 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.549355    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-234.ec2.internal.18d5e12432bb53e2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-234.ec2.internal,UID:ip-10-0-138-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-234.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-234.ec2.internal,},FirstTimestamp:2026-09-16 18:39:32.535849954 +0000 UTC m=+0.352615590,LastTimestamp:2026-09-16 18:39:32.535849954 +0000 UTC m=+0.352615590,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-234.ec2.internal,}"
Sep 16 18:39:32.556959 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.556902    2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-234.ec2.internal.18d5e12432bace6a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-234.ec2.internal.18d5e12432bace6a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-234.ec2.internal,UID:ip-10-0-138-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-234.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-234.ec2.internal,},FirstTimestamp:2026-09-16 18:39:32.535815786 +0000 UTC m=+0.352581364,LastTimestamp:2026-09-16 18:39:32.537750397 +0000 UTC m=+0.354515979,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-234.ec2.internal,}"
Sep 16 18:39:32.563353 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.563338    2575 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:39:32.563456 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.563396    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:39:32.563456 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.563411    2575 server.go:87] "Starting device plugin registration server"
Sep 16 18:39:32.564288 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.563727    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:39:32.564288 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.563746    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:39:32.564288 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.564035    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:39:32.564288 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.564118    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:39:32.564288 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.564127    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:39:32.564582 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.564567    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:39:32.564640 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.564617    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:32.565833 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.565685    2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-234.ec2.internal.18d5e12432bb1aa1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-234.ec2.internal.18d5e12432bb1aa1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-234.ec2.internal,UID:ip-10-0-138-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-234.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-234.ec2.internal,},FirstTimestamp:2026-09-16 18:39:32.535835297 +0000 UTC m=+0.352600874,LastTimestamp:2026-09-16 18:39:32.537771033 +0000 UTC m=+0.354536609,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-234.ec2.internal,}"
Sep 16 18:39:32.569084 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.569068    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xhbvn"
Sep 16 18:39:32.576915 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.576854    2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-234.ec2.internal.18d5e12432bb53e2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-234.ec2.internal.18d5e12432bb53e2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-234.ec2.internal,UID:ip-10-0-138-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-234.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-234.ec2.internal,},FirstTimestamp:2026-09-16 18:39:32.535849954 +0000 UTC m=+0.352615590,LastTimestamp:2026-09-16 18:39:32.537779466 +0000 UTC m=+0.354545043,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-234.ec2.internal,}"
Sep 16 18:39:32.590303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.578077    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xhbvn"
Sep 16 18:39:32.612742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.612705    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:39:32.613829 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.613810    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:39:32.613904 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.613854    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:39:32.613904 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.613871    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:39:32.614010 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.613933    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:39:32.616154 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.616127    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:32.664276 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.664257    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:32.664991 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.664978    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:32.665049 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.665002    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:32.665049 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.665013    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:32.665049 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.665034    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.674165 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.674151    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.674227 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.674170    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-234.ec2.internal\": node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:32.709598 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.709530    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:32.714600 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.714581    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal"]
Sep 16 18:39:32.714665 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.714655    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:32.715976 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.715960    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:32.716037 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.715987    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:32.716037 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.715996    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:32.717219 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.717207    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:32.717344 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.717331    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.717379 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.717359    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:32.717941 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.717922    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:32.717941 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.717938    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:32.718072 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.717952    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:32.718072 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.717959    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:32.718072 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.717967    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:32.718072 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.717973    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:32.719202 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.719188    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.719258 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.719213    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:32.719885 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.719872    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:32.719980 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.719894    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:32.719980 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.719903    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:32.720219 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.720201    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb49792980f6c23b527f95eef7d39f62-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal\" (UID: \"fb49792980f6c23b527f95eef7d39f62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.720301 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.720224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb49792980f6c23b527f95eef7d39f62-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal\" (UID: \"fb49792980f6c23b527f95eef7d39f62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.720301 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.720241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/96dd7b7104c0456a47b41f58b7878f94-config\") pod \"kube-apiserver-proxy-ip-10-0-138-234.ec2.internal\" (UID: \"96dd7b7104c0456a47b41f58b7878f94\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.748303 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.748285    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-234.ec2.internal\" not found" node="ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.752539 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.752523    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-234.ec2.internal\" not found" node="ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.809914 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.809897    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:32.821190 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.821168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb49792980f6c23b527f95eef7d39f62-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal\" (UID: \"fb49792980f6c23b527f95eef7d39f62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.821252 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.821193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb49792980f6c23b527f95eef7d39f62-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal\" (UID: \"fb49792980f6c23b527f95eef7d39f62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.821252 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.821208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/96dd7b7104c0456a47b41f58b7878f94-config\") pod \"kube-apiserver-proxy-ip-10-0-138-234.ec2.internal\" (UID: \"96dd7b7104c0456a47b41f58b7878f94\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.821316 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.821264    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/96dd7b7104c0456a47b41f58b7878f94-config\") pod \"kube-apiserver-proxy-ip-10-0-138-234.ec2.internal\" (UID: \"96dd7b7104c0456a47b41f58b7878f94\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.821316 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.821268    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb49792980f6c23b527f95eef7d39f62-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal\" (UID: \"fb49792980f6c23b527f95eef7d39f62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.821316 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:32.821287    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb49792980f6c23b527f95eef7d39f62-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal\" (UID: \"fb49792980f6c23b527f95eef7d39f62\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:32.910546 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:32.910509    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:33.010898 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:33.010850    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:33.051318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.051290    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:33.055330 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.055313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:33.111135 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:33.111109    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:33.211966 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:33.211596    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:33.313587 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:33.313525    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:33.417949 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:33.417924    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:33.460124 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.460096    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:39:33.460640 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.460258    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:39:33.518404 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:33.518378    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:33.550003 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.549979    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:33.551297 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.551263    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:33.551422 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.551306    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:33.551422 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.551322    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:33.583052 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.582992    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:34:33 +0000 UTC" deadline="2028-06-10 03:01:59.534759136 +0000 UTC"
Sep 16 18:39:33.583052 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.583025    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15176h22m25.951737467s"
Sep 16 18:39:33.618490 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:33.618464    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-234.ec2.internal\" not found"
Sep 16 18:39:33.624448 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:33.624402    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb49792980f6c23b527f95eef7d39f62.slice/crio-a78457dba1fd6ed9a8521e7ec468f7daec78558dcf70f62ea7e62720b72fa73f WatchSource:0}: Error finding container a78457dba1fd6ed9a8521e7ec468f7daec78558dcf70f62ea7e62720b72fa73f: Status 404 returned error can't find the container with id a78457dba1fd6ed9a8521e7ec468f7daec78558dcf70f62ea7e62720b72fa73f
Sep 16 18:39:33.624800 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:33.624763    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96dd7b7104c0456a47b41f58b7878f94.slice/crio-5482caa3cd249793f7005673ca98967204e34a60c1f0dc83a9e997d6766bc8c8 WatchSource:0}: Error finding container 5482caa3cd249793f7005673ca98967204e34a60c1f0dc83a9e997d6766bc8c8: Status 404 returned error can't find the container with id 5482caa3cd249793f7005673ca98967204e34a60c1f0dc83a9e997d6766bc8c8
Sep 16 18:39:33.629637 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.629620    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:39:33.647544 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.647526    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:33.676419 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.676398    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:33.720843 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.720824    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:33.734457 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.734441    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:33.736063 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.736051    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal"
Sep 16 18:39:33.749391 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:33.749376    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:34.074365 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.074288    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:34.343015 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.342937    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:34.499649 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.499617    2575 apiserver.go:52] "Watching apiserver"
Sep 16 18:39:34.506538 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.506512    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:39:34.508970 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.508941    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk","openshift-cluster-node-tuning-operator/tuned-ch22w","openshift-multus/multus-dkdhs","openshift-network-diagnostics/network-check-target-9bd2q","openshift-network-operator/iptables-alerter-xmmnv","kube-system/global-pull-secret-syncer-jjs7q","openshift-dns/node-resolver-ph95x","openshift-image-registry/node-ca-bvjjb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal","openshift-multus/multus-additional-cni-plugins-fgmrl","openshift-multus/network-metrics-daemon-7jzmn","openshift-ovn-kubernetes/ovnkube-node-jw2kd","kube-system/konnectivity-agent-nw2jb"]
Sep 16 18:39:34.510965 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.510940    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:34.511073 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.511051    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:34.512344 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.512326    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.513885 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.513863    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.515281 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.515259    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.516964 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.516941    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:34.517061 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.517024    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:34.518503 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.518478    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.519431 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.519414    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.519880    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.519948    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.520216    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.520583    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.520750    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.520989    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.521082    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v8lms\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.521191    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dqvhx\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.521196    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.521275    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.521321    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:39:34.521393 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.521374    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:34.522136 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.521198    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zzzlq\""
Sep 16 18:39:34.522136 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.521613    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:34.522136 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.521672    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:39:34.522136 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.522010    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:39:34.522136 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.522078    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:34.522431 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.522202    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:39:34.522431 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.522321    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgd4x\""
Sep 16 18:39:34.522431 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.522421    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:34.522604 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.522456    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:39:34.524856 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.524268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.525293 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.525274    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:39:34.525419 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.525276    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:39:34.525716 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.525694    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.526580 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.526349    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:39:34.526668 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.526593    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mm4l2\""
Sep 16 18:39:34.528548 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.528255    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.529636 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.529606    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:39:34.529636 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.529630    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mv4z6\""
Sep 16 18:39:34.531922 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.530069    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:39:34.531922 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.530338    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:39:34.531922 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.531109    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:39:34.531922 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.531618    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:39:34.532295 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-run-ovn\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.532295 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-modprobe-d\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.532405 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532329    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-run-openvswitch\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.532405 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-systemd-units\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.532506 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532421    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-sysctl-conf\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.532506 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-conf-dir\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.532506 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532479    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swshl\" (UniqueName: \"kubernetes.io/projected/ae87ca82-6c33-4412-87f9-92ba9f2858f3-kube-api-access-swshl\") pod \"node-ca-bvjjb\" (UID: \"ae87ca82-6c33-4412-87f9-92ba9f2858f3\") " pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.532664 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-sysctl-d\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.532664 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532544    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-sys\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.532664 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-host\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.532664 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-ovnkube-config\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.532664 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532635    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-lib-modules\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.532943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-socket-dir-parent\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.532943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-run-k8s-cni-cncf-io\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.532943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-var-lib-cni-multus\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.532943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgxv\" (UniqueName: \"kubernetes.io/projected/b386d3e2-ddd7-48d2-9c80-3bcf8004ab82-kube-api-access-xqgxv\") pod \"iptables-alerter-xmmnv\" (UID: \"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82\") " pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.532943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532824    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-cni-netd\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.532943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-env-overrides\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.532943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-tmp\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.532943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1328e944-9bd6-40cf-8418-ae2acebbc662-cni-binary-copy\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.532967    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-system-cni-dir\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-run-netns\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-daemon-config\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533093    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-log-socket\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mz6\" (UniqueName: \"kubernetes.io/projected/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-kube-api-access-h9mz6\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-etc-openvswitch\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533237    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-cni-bin\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-run\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533307    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-cnibin\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.533358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b386d3e2-ddd7-48d2-9c80-3bcf8004ab82-iptables-alerter-script\") pod \"iptables-alerter-xmmnv\" (UID: \"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82\") " pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b386d3e2-ddd7-48d2-9c80-3bcf8004ab82-host-slash\") pod \"iptables-alerter-xmmnv\" (UID: \"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82\") " pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae87ca82-6c33-4412-87f9-92ba9f2858f3-host\") pod \"node-ca-bvjjb\" (UID: \"ae87ca82-6c33-4412-87f9-92ba9f2858f3\") " pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae87ca82-6c33-4412-87f9-92ba9f2858f3-serviceca\") pod \"node-ca-bvjjb\" (UID: \"ae87ca82-6c33-4412-87f9-92ba9f2858f3\") " pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-kubelet-config\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533531    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-ovnkube-script-lib\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533560    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-systemd\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-etc-kubernetes\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533619    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-run-netns\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533650    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-var-lib-openvswitch\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533680    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-var-lib-cni-bin\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-kubelet\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533735    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:39:34.533990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533856    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6cmg\""
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534038    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534144    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534253    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfczg\""
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.533760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-run-systemd\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-run-ovn-kubernetes\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534362    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-hostroot\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-sysconfig\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-tuned\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-var-lib-kubelet\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km586\" (UniqueName: \"kubernetes.io/projected/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-kube-api-access-km586\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534536    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534546    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-node-log\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-slash\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.534621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534629    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-kubernetes\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-dbus\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534688    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-var-lib-kubelet\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-run-multus-certs\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534778    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx9nd\" (UniqueName: \"kubernetes.io/projected/1328e944-9bd6-40cf-8418-ae2acebbc662-kube-api-access-wx9nd\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534873    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-ovn-node-metrics-cert\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-cni-dir\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.534958    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.535087    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:34.535561 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.535154    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bqw9q\""
Sep 16 18:39:34.541629 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.541608    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:34.581384 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.581362    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h4grw"
Sep 16 18:39:34.594617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.594561    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h4grw"
Sep 16 18:39:34.619831 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.619782    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal" event={"ID":"96dd7b7104c0456a47b41f58b7878f94","Type":"ContainerStarted","Data":"5482caa3cd249793f7005673ca98967204e34a60c1f0dc83a9e997d6766bc8c8"}
Sep 16 18:39:34.620829 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.620807    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:39:34.621284 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.621263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal" event={"ID":"fb49792980f6c23b527f95eef7d39f62","Type":"ContainerStarted","Data":"a78457dba1fd6ed9a8521e7ec468f7daec78558dcf70f62ea7e62720b72fa73f"}
Sep 16 18:39:34.635108 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-sys-fs\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.635197 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635107    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psh5f\" (UniqueName: \"kubernetes.io/projected/dca28d1e-3d17-4eea-a864-5f984527ae0f-kube-api-access-psh5f\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:34.635197 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635167    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f6c1461d-bc9d-4506-a352-f700080df9e6-konnectivity-ca\") pod \"konnectivity-agent-nw2jb\" (UID: \"f6c1461d-bc9d-4506-a352-f700080df9e6\") " pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:34.635322 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06d59a6-6b82-4c92-adc3-c579ab24cffd-tmp-dir\") pod \"node-resolver-ph95x\" (UID: \"e06d59a6-6b82-4c92-adc3-c579ab24cffd\") " pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.635322 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635242    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be02496-7563-49b9-90a6-adcce7c4037e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.635322 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635275    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-sysconfig\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.635322 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-tuned\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be02496-7563-49b9-90a6-adcce7c4037e-cnibin\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635349    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-sysconfig\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-var-lib-kubelet\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km586\" (UniqueName: \"kubernetes.io/projected/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-kube-api-access-km586\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635420    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-var-lib-kubelet\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635422    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-node-log\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635459    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-node-log\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635457    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635506    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-slash\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.635523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635527    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-kubernetes\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-dbus\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635579    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-var-lib-kubelet\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-run-multus-certs\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9nd\" (UniqueName: \"kubernetes.io/projected/1328e944-9bd6-40cf-8418-ae2acebbc662-kube-api-access-wx9nd\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635671    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-kubernetes\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-ovn-node-metrics-cert\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-cni-dir\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsvt\" (UniqueName: \"kubernetes.io/projected/e06d59a6-6b82-4c92-adc3-c579ab24cffd-kube-api-access-wjsvt\") pod \"node-resolver-ph95x\" (UID: \"e06d59a6-6b82-4c92-adc3-c579ab24cffd\") " pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-run-ovn\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-modprobe-d\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635819    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-run-openvswitch\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-systemd-units\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635853    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-dbus\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635852    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-slash\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-sysctl-conf\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-conf-dir\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-sysctl-conf\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swshl\" (UniqueName: \"kubernetes.io/projected/ae87ca82-6c33-4412-87f9-92ba9f2858f3-kube-api-access-swshl\") pod \"node-ca-bvjjb\" (UID: \"ae87ca82-6c33-4412-87f9-92ba9f2858f3\") " pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636010    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-run-ovn\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635969    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-cni-dir\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636029    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-modprobe-d\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636030    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6be02496-7563-49b9-90a6-adcce7c4037e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636036    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-conf-dir\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636038    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-systemd-units\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-run-openvswitch\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-sysctl-d\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-sys\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636105    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-var-lib-kubelet\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.635561    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-host\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635906    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-run-multus-certs\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.636963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be02496-7563-49b9-90a6-adcce7c4037e-system-cni-dir\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.636177    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret podName:fd64fe4d-7ba3-4355-84b8-8df3b98703c4 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:35.136156306 +0000 UTC m=+2.952921887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret") pod "global-pull-secret-syncer-jjs7q" (UID: "fd64fe4d-7ba3-4355-84b8-8df3b98703c4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-host\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-ovnkube-config\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636268    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-sysctl-d\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-lib-modules\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-socket-dir-parent\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636323    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-sys\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-run-k8s-cni-cncf-io\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-var-lib-cni-multus\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-socket-dir-parent\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636420    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-run-k8s-cni-cncf-io\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636427    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-lib-modules\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636422    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgxv\" (UniqueName: \"kubernetes.io/projected/b386d3e2-ddd7-48d2-9c80-3bcf8004ab82-kube-api-access-xqgxv\") pod \"iptables-alerter-xmmnv\" (UID: \"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82\") " pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636494    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-var-lib-cni-multus\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.635732    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636551    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-device-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.637855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e06d59a6-6b82-4c92-adc3-c579ab24cffd-hosts-file\") pod \"node-resolver-ph95x\" (UID: \"e06d59a6-6b82-4c92-adc3-c579ab24cffd\") " pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-cni-netd\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-env-overrides\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-tmp\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636722    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-cni-netd\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1328e944-9bd6-40cf-8418-ae2acebbc662-cni-binary-copy\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-system-cni-dir\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-run-netns\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-daemon-config\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-ovnkube-config\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636856    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-log-socket\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mz6\" (UniqueName: \"kubernetes.io/projected/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-kube-api-access-h9mz6\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-etc-openvswitch\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636926    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-run-netns\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-cni-bin\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636986    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-system-cni-dir\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.636998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-run\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.638742 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-cnibin\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b386d3e2-ddd7-48d2-9c80-3bcf8004ab82-iptables-alerter-script\") pod \"iptables-alerter-xmmnv\" (UID: \"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82\") " pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b386d3e2-ddd7-48d2-9c80-3bcf8004ab82-host-slash\") pod \"iptables-alerter-xmmnv\" (UID: \"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82\") " pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae87ca82-6c33-4412-87f9-92ba9f2858f3-host\") pod \"node-ca-bvjjb\" (UID: \"ae87ca82-6c33-4412-87f9-92ba9f2858f3\") " pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae87ca82-6c33-4412-87f9-92ba9f2858f3-serviceca\") pod \"node-ca-bvjjb\" (UID: \"ae87ca82-6c33-4412-87f9-92ba9f2858f3\") " pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-socket-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-kubelet-config\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-ovnkube-script-lib\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637213    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-cnibin\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637228    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-log-socket\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae87ca82-6c33-4412-87f9-92ba9f2858f3-host\") pod \"node-ca-bvjjb\" (UID: \"ae87ca82-6c33-4412-87f9-92ba9f2858f3\") " pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637243    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f6c1461d-bc9d-4506-a352-f700080df9e6-agent-certs\") pod \"konnectivity-agent-nw2jb\" (UID: \"f6c1461d-bc9d-4506-a352-f700080df9e6\") " pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mbqv\" (UniqueName: \"kubernetes.io/projected/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-kube-api-access-5mbqv\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637281    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-cni-bin\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637283    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1328e944-9bd6-40cf-8418-ae2acebbc662-cni-binary-copy\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be02496-7563-49b9-90a6-adcce7c4037e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9w6\" (UniqueName: \"kubernetes.io/projected/6be02496-7563-49b9-90a6-adcce7c4037e-kube-api-access-rn9w6\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.639617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637346    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-run\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637371    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-systemd\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637398    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-etc-kubernetes\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-registration-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6be02496-7563-49b9-90a6-adcce7c4037e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637445    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-etc-openvswitch\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637627    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-run-netns\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-var-lib-openvswitch\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637690    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-ovnkube-script-lib\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-env-overrides\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-var-lib-cni-bin\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637753    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1328e944-9bd6-40cf-8418-ae2acebbc662-multus-daemon-config\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-kubelet\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-run-systemd\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-run-ovn-kubernetes\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637856    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-systemd\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.640385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-hostroot\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637899    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-host-var-lib-cni-bin\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-run-systemd\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637952    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b386d3e2-ddd7-48d2-9c80-3bcf8004ab82-host-slash\") pod \"iptables-alerter-xmmnv\" (UID: \"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82\") " pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.637981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-kubelet-config\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.638000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-run-ovn-kubernetes\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.638014    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-kubelet\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.638022    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-host-run-netns\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.638055    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-hostroot\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.638058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-var-lib-openvswitch\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.638067    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1328e944-9bd6-40cf-8418-ae2acebbc662-etc-kubernetes\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.638108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ae87ca82-6c33-4412-87f9-92ba9f2858f3-serviceca\") pod \"node-ca-bvjjb\" (UID: \"ae87ca82-6c33-4412-87f9-92ba9f2858f3\") " pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.638179    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b386d3e2-ddd7-48d2-9c80-3bcf8004ab82-iptables-alerter-script\") pod \"iptables-alerter-xmmnv\" (UID: \"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82\") " pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.640761    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-tmp\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.640841    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-etc-tuned\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.641303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.641003    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-ovn-node-metrics-cert\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.661597 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.661570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km586\" (UniqueName: \"kubernetes.io/projected/e6a884b5-5dcb-4d4c-99f3-19bbc6843b37-kube-api-access-km586\") pod \"tuned-ch22w\" (UID: \"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37\") " pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.667799 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.667752    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:34.667799 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.667776    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:34.667799 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.667789    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kr2tz for pod openshift-network-diagnostics/network-check-target-9bd2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:34.667986 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.667853    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz podName:c0e375bf-00f2-427d-a9e8-00b934574883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:35.167835076 +0000 UTC m=+2.984600645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kr2tz" (UniqueName: "kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz") pod "network-check-target-9bd2q" (UID: "c0e375bf-00f2-427d-a9e8-00b934574883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:34.673013 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.672991    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swshl\" (UniqueName: \"kubernetes.io/projected/ae87ca82-6c33-4412-87f9-92ba9f2858f3-kube-api-access-swshl\") pod \"node-ca-bvjjb\" (UID: \"ae87ca82-6c33-4412-87f9-92ba9f2858f3\") " pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.673279 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.673256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx9nd\" (UniqueName: \"kubernetes.io/projected/1328e944-9bd6-40cf-8418-ae2acebbc662-kube-api-access-wx9nd\") pod \"multus-dkdhs\" (UID: \"1328e944-9bd6-40cf-8418-ae2acebbc662\") " pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.674068 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.674027    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mz6\" (UniqueName: \"kubernetes.io/projected/c0a6e8ff-376d-457f-83d7-b1eb38ccb726-kube-api-access-h9mz6\") pod \"ovnkube-node-jw2kd\" (UID: \"c0a6e8ff-376d-457f-83d7-b1eb38ccb726\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.675100 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.675070    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgxv\" (UniqueName: \"kubernetes.io/projected/b386d3e2-ddd7-48d2-9c80-3bcf8004ab82-kube-api-access-xqgxv\") pod \"iptables-alerter-xmmnv\" (UID: \"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82\") " pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.739017 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.738991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06d59a6-6b82-4c92-adc3-c579ab24cffd-tmp-dir\") pod \"node-resolver-ph95x\" (UID: \"e06d59a6-6b82-4c92-adc3-c579ab24cffd\") " pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.739121 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739037    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be02496-7563-49b9-90a6-adcce7c4037e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739121 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be02496-7563-49b9-90a6-adcce7c4037e-cnibin\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739121 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739105    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsvt\" (UniqueName: \"kubernetes.io/projected/e06d59a6-6b82-4c92-adc3-c579ab24cffd-kube-api-access-wjsvt\") pod \"node-resolver-ph95x\" (UID: \"e06d59a6-6b82-4c92-adc3-c579ab24cffd\") " pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739144    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6be02496-7563-49b9-90a6-adcce7c4037e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739171    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be02496-7563-49b9-90a6-adcce7c4037e-system-cni-dir\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-device-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e06d59a6-6b82-4c92-adc3-c579ab24cffd-hosts-file\") pod \"node-resolver-ph95x\" (UID: \"e06d59a6-6b82-4c92-adc3-c579ab24cffd\") " pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739226    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be02496-7563-49b9-90a6-adcce7c4037e-cnibin\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06d59a6-6b82-4c92-adc3-c579ab24cffd-tmp-dir\") pod \"node-resolver-ph95x\" (UID: \"e06d59a6-6b82-4c92-adc3-c579ab24cffd\") " pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739231    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739285    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be02496-7563-49b9-90a6-adcce7c4037e-system-cni-dir\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be02496-7563-49b9-90a6-adcce7c4037e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739311    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-device-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739333    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-socket-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739344    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e06d59a6-6b82-4c92-adc3-c579ab24cffd-hosts-file\") pod \"node-resolver-ph95x\" (UID: \"e06d59a6-6b82-4c92-adc3-c579ab24cffd\") " pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f6c1461d-bc9d-4506-a352-f700080df9e6-agent-certs\") pod \"konnectivity-agent-nw2jb\" (UID: \"f6c1461d-bc9d-4506-a352-f700080df9e6\") " pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mbqv\" (UniqueName: \"kubernetes.io/projected/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-kube-api-access-5mbqv\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be02496-7563-49b9-90a6-adcce7c4037e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9w6\" (UniqueName: \"kubernetes.io/projected/6be02496-7563-49b9-90a6-adcce7c4037e-kube-api-access-rn9w6\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739447    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-socket-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-registration-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6be02496-7563-49b9-90a6-adcce7c4037e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739551    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-sys-fs\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739599    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psh5f\" (UniqueName: \"kubernetes.io/projected/dca28d1e-3d17-4eea-a864-5f984527ae0f-kube-api-access-psh5f\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:34.739825 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f6c1461d-bc9d-4506-a352-f700080df9e6-konnectivity-ca\") pod \"konnectivity-agent-nw2jb\" (UID: \"f6c1461d-bc9d-4506-a352-f700080df9e6\") " pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:34.740446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739938    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-registration-dir\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.740446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739950    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.740446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.739982    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be02496-7563-49b9-90a6-adcce7c4037e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.740446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.740031    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-sys-fs\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.740446 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.740095    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:34.740446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.740106    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6be02496-7563-49b9-90a6-adcce7c4037e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.740446 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:34.740175    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs podName:dca28d1e-3d17-4eea-a864-5f984527ae0f nodeName:}" failed. No retries permitted until 2026-09-16 18:39:35.24015683 +0000 UTC m=+3.056922401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs") pod "network-metrics-daemon-7jzmn" (UID: "dca28d1e-3d17-4eea-a864-5f984527ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:34.740446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.740294    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f6c1461d-bc9d-4506-a352-f700080df9e6-konnectivity-ca\") pod \"konnectivity-agent-nw2jb\" (UID: \"f6c1461d-bc9d-4506-a352-f700080df9e6\") " pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:34.740889 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.740464    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6be02496-7563-49b9-90a6-adcce7c4037e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.741997 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.741975    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f6c1461d-bc9d-4506-a352-f700080df9e6-agent-certs\") pod \"konnectivity-agent-nw2jb\" (UID: \"f6c1461d-bc9d-4506-a352-f700080df9e6\") " pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:34.760091 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.760070    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mbqv\" (UniqueName: \"kubernetes.io/projected/c53f01d5-60d7-41d0-8d92-c4bd5dcd1645-kube-api-access-5mbqv\") pod \"aws-ebs-csi-driver-node-q5slk\" (UID: \"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.760245 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.760165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsvt\" (UniqueName: \"kubernetes.io/projected/e06d59a6-6b82-4c92-adc3-c579ab24cffd-kube-api-access-wjsvt\") pod \"node-resolver-ph95x\" (UID: \"e06d59a6-6b82-4c92-adc3-c579ab24cffd\") " pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.763766 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.763731    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psh5f\" (UniqueName: \"kubernetes.io/projected/dca28d1e-3d17-4eea-a864-5f984527ae0f-kube-api-access-psh5f\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:34.763876 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.763856    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9w6\" (UniqueName: \"kubernetes.io/projected/6be02496-7563-49b9-90a6-adcce7c4037e-kube-api-access-rn9w6\") pod \"multus-additional-cni-plugins-fgmrl\" (UID: \"6be02496-7563-49b9-90a6-adcce7c4037e\") " pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:34.829494 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.829463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:34.835350 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.835327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ch22w"
Sep 16 18:39:34.846016 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.845956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dkdhs"
Sep 16 18:39:34.852435 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.852413    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xmmnv"
Sep 16 18:39:34.861282 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.861260    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bvjjb"
Sep 16 18:39:34.871120 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.871099    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:34.876075 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.876043    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk"
Sep 16 18:39:34.884126 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.884105    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ph95x"
Sep 16 18:39:34.889985 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:34.889964    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fgmrl"
Sep 16 18:39:35.142412 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:35.142320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:35.142562 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:35.142445    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:35.142562 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:35.142511    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret podName:fd64fe4d-7ba3-4355-84b8-8df3b98703c4 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:36.142492545 +0000 UTC m=+3.959258128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret") pod "global-pull-secret-syncer-jjs7q" (UID: "fd64fe4d-7ba3-4355-84b8-8df3b98703c4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:35.243705 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:35.243667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:35.243895 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:35.243795    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:35.243895 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:35.243863    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:35.243989 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:35.243930    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:35.243989 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:35.243947    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:35.243989 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:35.243952    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs podName:dca28d1e-3d17-4eea-a864-5f984527ae0f nodeName:}" failed. No retries permitted until 2026-09-16 18:39:36.243928341 +0000 UTC m=+4.060693925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs") pod "network-metrics-daemon-7jzmn" (UID: "dca28d1e-3d17-4eea-a864-5f984527ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:35.243989 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:35.243958    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kr2tz for pod openshift-network-diagnostics/network-check-target-9bd2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:35.244136 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:35.244002    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz podName:c0e375bf-00f2-427d-a9e8-00b934574883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:36.243988917 +0000 UTC m=+4.060754488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr2tz" (UniqueName: "kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz") pod "network-check-target-9bd2q" (UID: "c0e375bf-00f2-427d-a9e8-00b934574883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:35.597772 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:35.597666    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:35 +0000 UTC" deadline="2028-03-15 12:05:58.120110022 +0000 UTC"
Sep 16 18:39:35.597772 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:35.597702    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13097h26m22.522411936s"
Sep 16 18:39:35.615241 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:35.615214    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:35.615339 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:35.615320    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:35.650806 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:35.650766    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1328e944_9bd6_40cf_8418_ae2acebbc662.slice/crio-c30665c4a6d40773fd6e4b5a09a46592490a161a568f1febc5a36d3a2c33d3e7 WatchSource:0}: Error finding container c30665c4a6d40773fd6e4b5a09a46592490a161a568f1febc5a36d3a2c33d3e7: Status 404 returned error can't find the container with id c30665c4a6d40773fd6e4b5a09a46592490a161a568f1febc5a36d3a2c33d3e7
Sep 16 18:39:35.652198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:35.652172    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53f01d5_60d7_41d0_8d92_c4bd5dcd1645.slice/crio-a126bd56c0daf91ddd722aef198f67bea4bee945cab69260df6f141a77654f56 WatchSource:0}: Error finding container a126bd56c0daf91ddd722aef198f67bea4bee945cab69260df6f141a77654f56: Status 404 returned error can't find the container with id a126bd56c0daf91ddd722aef198f67bea4bee945cab69260df6f141a77654f56
Sep 16 18:39:35.656289 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:35.656265    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be02496_7563_49b9_90a6_adcce7c4037e.slice/crio-884b1a5599d05a6ace6c8aa1ef142288bf505bb531991284f6bd832858cc81d9 WatchSource:0}: Error finding container 884b1a5599d05a6ace6c8aa1ef142288bf505bb531991284f6bd832858cc81d9: Status 404 returned error can't find the container with id 884b1a5599d05a6ace6c8aa1ef142288bf505bb531991284f6bd832858cc81d9
Sep 16 18:39:35.657162 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:35.657138    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a6e8ff_376d_457f_83d7_b1eb38ccb726.slice/crio-9b92cf8b8780f910b508f86d12a717d6e479750d7dae893123028911ab893809 WatchSource:0}: Error finding container 9b92cf8b8780f910b508f86d12a717d6e479750d7dae893123028911ab893809: Status 404 returned error can't find the container with id 9b92cf8b8780f910b508f86d12a717d6e479750d7dae893123028911ab893809
Sep 16 18:39:35.657962 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:35.657942    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb386d3e2_ddd7_48d2_9c80_3bcf8004ab82.slice/crio-df6ee28aab1d2e7d14cc93502e5c9104df6596b43c6150a3f93ef1f38733a48d WatchSource:0}: Error finding container df6ee28aab1d2e7d14cc93502e5c9104df6596b43c6150a3f93ef1f38733a48d: Status 404 returned error can't find the container with id df6ee28aab1d2e7d14cc93502e5c9104df6596b43c6150a3f93ef1f38733a48d
Sep 16 18:39:35.659221 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:35.659198    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c1461d_bc9d_4506_a352_f700080df9e6.slice/crio-4694ba647c56c0e9cf7999180ede57b32327a7a7bbdf32944cb7ebe100c0db07 WatchSource:0}: Error finding container 4694ba647c56c0e9cf7999180ede57b32327a7a7bbdf32944cb7ebe100c0db07: Status 404 returned error can't find the container with id 4694ba647c56c0e9cf7999180ede57b32327a7a7bbdf32944cb7ebe100c0db07
Sep 16 18:39:35.660893 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:35.660512    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae87ca82_6c33_4412_87f9_92ba9f2858f3.slice/crio-e10314fc8a0dad0b830e3d602ae8ac70149865649d3f70a2075f2736b825b5c2 WatchSource:0}: Error finding container e10314fc8a0dad0b830e3d602ae8ac70149865649d3f70a2075f2736b825b5c2: Status 404 returned error can't find the container with id e10314fc8a0dad0b830e3d602ae8ac70149865649d3f70a2075f2736b825b5c2
Sep 16 18:39:35.661235 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:39:35.660984    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a884b5_5dcb_4d4c_99f3_19bbc6843b37.slice/crio-df6d3290e9b9aea8ebc3d8427088d5eded9503ee4a0eec32b804bb9bba9902ae WatchSource:0}: Error finding container df6d3290e9b9aea8ebc3d8427088d5eded9503ee4a0eec32b804bb9bba9902ae: Status 404 returned error can't find the container with id df6d3290e9b9aea8ebc3d8427088d5eded9503ee4a0eec32b804bb9bba9902ae
Sep 16 18:39:36.153701 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.152745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:36.153701 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.153153    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:36.153701 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.153228    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret podName:fd64fe4d-7ba3-4355-84b8-8df3b98703c4 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:38.153205093 +0000 UTC m=+5.969970661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret") pod "global-pull-secret-syncer-jjs7q" (UID: "fd64fe4d-7ba3-4355-84b8-8df3b98703c4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:36.255272 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.254381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:36.255272 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.254443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:36.255272 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.254609    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:36.255272 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.254674    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs podName:dca28d1e-3d17-4eea-a864-5f984527ae0f nodeName:}" failed. No retries permitted until 2026-09-16 18:39:38.254654142 +0000 UTC m=+6.071419713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs") pod "network-metrics-daemon-7jzmn" (UID: "dca28d1e-3d17-4eea-a864-5f984527ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:36.255272 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.255150    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:36.255272 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.255169    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:36.255272 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.255182    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kr2tz for pod openshift-network-diagnostics/network-check-target-9bd2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:36.255272 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.255231    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz podName:c0e375bf-00f2-427d-a9e8-00b934574883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:38.255214842 +0000 UTC m=+6.071980409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr2tz" (UniqueName: "kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz") pod "network-check-target-9bd2q" (UID: "c0e375bf-00f2-427d-a9e8-00b934574883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:36.598382 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.598239    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:35 +0000 UTC" deadline="2028-06-19 11:21:55.605456828 +0000 UTC"
Sep 16 18:39:36.598382 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.598297    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15400h42m19.007164302s"
Sep 16 18:39:36.617185 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.616458    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:36.617185 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.616588    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:36.617185 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.617002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:36.617185 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:36.617115    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:36.637471 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.637390    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ph95x" event={"ID":"e06d59a6-6b82-4c92-adc3-c579ab24cffd","Type":"ContainerStarted","Data":"86cd5e3761c0a96329caa9eb2ca5cd366efa4e9857aaf8995e93499d3cc223d6"}
Sep 16 18:39:36.642815 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.642785    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xmmnv" event={"ID":"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82","Type":"ContainerStarted","Data":"df6ee28aab1d2e7d14cc93502e5c9104df6596b43c6150a3f93ef1f38733a48d"}
Sep 16 18:39:36.645475 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.645410    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk" event={"ID":"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645","Type":"ContainerStarted","Data":"a126bd56c0daf91ddd722aef198f67bea4bee945cab69260df6f141a77654f56"}
Sep 16 18:39:36.649265 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.649238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bvjjb" event={"ID":"ae87ca82-6c33-4412-87f9-92ba9f2858f3","Type":"ContainerStarted","Data":"e10314fc8a0dad0b830e3d602ae8ac70149865649d3f70a2075f2736b825b5c2"}
Sep 16 18:39:36.653801 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.653763    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ch22w" event={"ID":"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37","Type":"ContainerStarted","Data":"df6d3290e9b9aea8ebc3d8427088d5eded9503ee4a0eec32b804bb9bba9902ae"}
Sep 16 18:39:36.663514 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.663487    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nw2jb" event={"ID":"f6c1461d-bc9d-4506-a352-f700080df9e6","Type":"ContainerStarted","Data":"4694ba647c56c0e9cf7999180ede57b32327a7a7bbdf32944cb7ebe100c0db07"}
Sep 16 18:39:36.669195 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.669169    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" event={"ID":"c0a6e8ff-376d-457f-83d7-b1eb38ccb726","Type":"ContainerStarted","Data":"9b92cf8b8780f910b508f86d12a717d6e479750d7dae893123028911ab893809"}
Sep 16 18:39:36.678225 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.678190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgmrl" event={"ID":"6be02496-7563-49b9-90a6-adcce7c4037e","Type":"ContainerStarted","Data":"884b1a5599d05a6ace6c8aa1ef142288bf505bb531991284f6bd832858cc81d9"}
Sep 16 18:39:36.683356 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.683329    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dkdhs" event={"ID":"1328e944-9bd6-40cf-8418-ae2acebbc662","Type":"ContainerStarted","Data":"c30665c4a6d40773fd6e4b5a09a46592490a161a568f1febc5a36d3a2c33d3e7"}
Sep 16 18:39:36.696496 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.696469    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal" event={"ID":"96dd7b7104c0456a47b41f58b7878f94","Type":"ContainerStarted","Data":"79d614c0e8423847d910f35e233802c6f9a70d3538f803fecf7eb2b0fe5b9553"}
Sep 16 18:39:36.717715 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.717687    2575 generic.go:356] "Generic (PLEG): container finished" podID="fb49792980f6c23b527f95eef7d39f62" containerID="f0fa6ba1920e8c7762ef6b75cc4d417e13227fad2e6371ca8e32f63df365ab1a" exitCode=0
Sep 16 18:39:36.717845 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.717727    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal" event={"ID":"fb49792980f6c23b527f95eef7d39f62","Type":"ContainerDied","Data":"f0fa6ba1920e8c7762ef6b75cc4d417e13227fad2e6371ca8e32f63df365ab1a"}
Sep 16 18:39:36.742643 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:36.742598    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.ec2.internal" podStartSLOduration=2.742580148 podStartE2EDuration="2.742580148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:36.717508336 +0000 UTC m=+4.534273927" watchObservedRunningTime="2026-09-16 18:39:36.742580148 +0000 UTC m=+4.559345741"
Sep 16 18:39:37.614646 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:37.614612    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:37.615337 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:37.614765    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:37.747040 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:37.746957    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal" event={"ID":"fb49792980f6c23b527f95eef7d39f62","Type":"ContainerStarted","Data":"b10f15c01728612c0cbf6b8a06388a5e1ea1cac4012b14c68e5690d466a6c424"}
Sep 16 18:39:37.776073 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:37.772874    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.ec2.internal" podStartSLOduration=3.772853213 podStartE2EDuration="3.772853213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:37.771728782 +0000 UTC m=+5.588494372" watchObservedRunningTime="2026-09-16 18:39:37.772853213 +0000 UTC m=+5.589618803"
Sep 16 18:39:38.171375 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:38.171289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:38.171558 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.171468    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:38.171558 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.171545    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret podName:fd64fe4d-7ba3-4355-84b8-8df3b98703c4 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:42.17152539 +0000 UTC m=+9.988290970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret") pod "global-pull-secret-syncer-jjs7q" (UID: "fd64fe4d-7ba3-4355-84b8-8df3b98703c4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:38.272002 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:38.271967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:38.272192 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:38.272025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:38.272192 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.272171    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:38.272310 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.272233    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs podName:dca28d1e-3d17-4eea-a864-5f984527ae0f nodeName:}" failed. No retries permitted until 2026-09-16 18:39:42.272214926 +0000 UTC m=+10.088980497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs") pod "network-metrics-daemon-7jzmn" (UID: "dca28d1e-3d17-4eea-a864-5f984527ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:38.272667 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.272648    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:38.272737 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.272679    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:38.272737 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.272695    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kr2tz for pod openshift-network-diagnostics/network-check-target-9bd2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:38.272871 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.272746    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz podName:c0e375bf-00f2-427d-a9e8-00b934574883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:42.272729854 +0000 UTC m=+10.089495422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr2tz" (UniqueName: "kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz") pod "network-check-target-9bd2q" (UID: "c0e375bf-00f2-427d-a9e8-00b934574883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:38.614969 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:38.614890    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:38.615461 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.615056    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:38.615571 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:38.615551    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:38.615700 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:38.615655    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:39.614890 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:39.614845    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:39.615096 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:39.614990    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:40.614260 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:40.614176    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:40.614513 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:40.614330    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:40.614937 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:40.614745    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:40.614937 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:40.614868    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:41.614992 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:41.614908    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:41.615404 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:41.615087    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:42.207676 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:42.207637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:42.207879 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.207815    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:42.207879 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.207878    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret podName:fd64fe4d-7ba3-4355-84b8-8df3b98703c4 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:50.20786175 +0000 UTC m=+18.024627315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret") pod "global-pull-secret-syncer-jjs7q" (UID: "fd64fe4d-7ba3-4355-84b8-8df3b98703c4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:42.308733 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:42.308690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:42.308918 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:42.308798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:42.308918 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.308879    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:42.309026 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.308927    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:42.309026 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.308945    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:42.309026 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.308954    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs podName:dca28d1e-3d17-4eea-a864-5f984527ae0f nodeName:}" failed. No retries permitted until 2026-09-16 18:39:50.308933216 +0000 UTC m=+18.125698793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs") pod "network-metrics-daemon-7jzmn" (UID: "dca28d1e-3d17-4eea-a864-5f984527ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:42.309026 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.308957    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kr2tz for pod openshift-network-diagnostics/network-check-target-9bd2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:42.309026 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.309010    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz podName:c0e375bf-00f2-427d-a9e8-00b934574883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:50.308994341 +0000 UTC m=+18.125759919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr2tz" (UniqueName: "kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz") pod "network-check-target-9bd2q" (UID: "c0e375bf-00f2-427d-a9e8-00b934574883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:42.616514 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:42.615893    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:42.616514 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.616010    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:42.616514 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:42.616342    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:42.616514 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:42.616426    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:43.614267 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:43.614224    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:43.614430 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:43.614365    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:44.614776 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:44.614741    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:44.615224 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:44.614868    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:44.615224 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:44.614926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:44.615224 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:44.615052    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:45.614100 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:45.614064    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:45.614293 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:45.614188    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:46.615371 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:46.615046    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:46.615371 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:46.615198    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:46.615371 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:46.615238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:46.615371 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:46.615315    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:47.614340 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:47.614314    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:47.614499 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:47.614409    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:48.614286 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:48.614157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:48.614286 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:48.614157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:48.614803 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:48.614287    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:48.614803 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:48.614365    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:49.614659 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:49.614494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:49.615087 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:49.614742    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:50.267477 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:50.267447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:50.267678 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.267569    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:50.267678 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.267635    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret podName:fd64fe4d-7ba3-4355-84b8-8df3b98703c4 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:06.267617421 +0000 UTC m=+34.084382984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret") pod "global-pull-secret-syncer-jjs7q" (UID: "fd64fe4d-7ba3-4355-84b8-8df3b98703c4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:50.368001 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:50.367961    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:50.368172 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:50.368057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:50.368172 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.368139    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:50.368267 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.368189    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:50.368267 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.368211    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:50.368267 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.368224    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kr2tz for pod openshift-network-diagnostics/network-check-target-9bd2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:50.368267 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.368225    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs podName:dca28d1e-3d17-4eea-a864-5f984527ae0f nodeName:}" failed. No retries permitted until 2026-09-16 18:40:06.368203842 +0000 UTC m=+34.184969426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs") pod "network-metrics-daemon-7jzmn" (UID: "dca28d1e-3d17-4eea-a864-5f984527ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:50.368452 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.368288    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz podName:c0e375bf-00f2-427d-a9e8-00b934574883 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:06.368270651 +0000 UTC m=+34.185036221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr2tz" (UniqueName: "kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz") pod "network-check-target-9bd2q" (UID: "c0e375bf-00f2-427d-a9e8-00b934574883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:50.614419 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:50.614379    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:50.614649 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:50.614379    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:50.614649 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.614531    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:50.614649 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:50.614569    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:51.614586 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:51.614544    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:51.614782 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:51.614670    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:52.614982 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:52.614949    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:52.615424 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:52.615070    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:52.615424 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:52.615109    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:52.615424 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:52.615174    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:53.614526 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.614374    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:53.614526 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:53.614488    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:53.787300 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.787095    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ch22w" event={"ID":"e6a884b5-5dcb-4d4c-99f3-19bbc6843b37","Type":"ContainerStarted","Data":"66519019a42569b161a32a1e42eef382b86107873de9d916e19ae64e05890dec"}
Sep 16 18:39:53.788976 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.788950    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nw2jb" event={"ID":"f6c1461d-bc9d-4506-a352-f700080df9e6","Type":"ContainerStarted","Data":"c7b8ae41c57e33f9182bc021cbca5deb5fda9479b7186036bf0257fa34acdff6"}
Sep 16 18:39:53.791149 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.791120    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" event={"ID":"c0a6e8ff-376d-457f-83d7-b1eb38ccb726","Type":"ContainerStarted","Data":"af5cfcf8c5feda8f894d161350bb0d09fb95688b3f33410222892280ab998d33"}
Sep 16 18:39:53.791255 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.791205    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" event={"ID":"c0a6e8ff-376d-457f-83d7-b1eb38ccb726","Type":"ContainerStarted","Data":"2f810630b5db338494187b572e7b838d6e2a08ba038abf457493ae7fc04e6658"}
Sep 16 18:39:53.791312 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.791254    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" event={"ID":"c0a6e8ff-376d-457f-83d7-b1eb38ccb726","Type":"ContainerStarted","Data":"d0ecc78a442ffbd505af69993087de2e27df94a27d1f47a242560e02d582e75a"}
Sep 16 18:39:53.792615 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.792589    2575 generic.go:356] "Generic (PLEG): container finished" podID="6be02496-7563-49b9-90a6-adcce7c4037e" containerID="b1940e417a24676913f92998e28df7654a188ab2a584c3696b5b9f6785296c7c" exitCode=0
Sep 16 18:39:53.792724 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.792681    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgmrl" event={"ID":"6be02496-7563-49b9-90a6-adcce7c4037e","Type":"ContainerDied","Data":"b1940e417a24676913f92998e28df7654a188ab2a584c3696b5b9f6785296c7c"}
Sep 16 18:39:53.794283 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.794245    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dkdhs" event={"ID":"1328e944-9bd6-40cf-8418-ae2acebbc662","Type":"ContainerStarted","Data":"fa0728fa963fe94c8d823b6c9aa8fb08c7e74b7bb14c63a9ef54ee14a9e4559c"}
Sep 16 18:39:53.799268 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.796495    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ph95x" event={"ID":"e06d59a6-6b82-4c92-adc3-c579ab24cffd","Type":"ContainerStarted","Data":"311e94eb72e5755bbd0165db38600d184baa7b5571e07b43af9a2984a760d2e5"}
Sep 16 18:39:53.800811 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.800791    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk" event={"ID":"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645","Type":"ContainerStarted","Data":"9df7c8c0f2a71ded6392da8d60fe8b4306e656f099f4ec9f5b5b145e1e8fc725"}
Sep 16 18:39:53.802383 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.802361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bvjjb" event={"ID":"ae87ca82-6c33-4412-87f9-92ba9f2858f3","Type":"ContainerStarted","Data":"cd8c086c386f81f798f6a70e495a51f44ed1a3b051bae18eebbc1ad250689f44"}
Sep 16 18:39:53.814701 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.814648    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ch22w" podStartSLOduration=3.060766167 podStartE2EDuration="20.814638192s" totalImagesPullingTime="17.753872025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:53.813865901 +0000 UTC m=+21.630631487" watchObservedRunningTime="2026-09-16 18:39:53.814638192 +0000 UTC m=+21.631403778"
Sep 16 18:39:53.852280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.852207    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dkdhs" podStartSLOduration=3.028368143 podStartE2EDuration="20.852192278s" totalImagesPullingTime="17.823824135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:53.851981168 +0000 UTC m=+21.668746755" watchObservedRunningTime="2026-09-16 18:39:53.852192278 +0000 UTC m=+21.668957863"
Sep 16 18:39:53.953364 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.953324    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nw2jb" podStartSLOduration=3.198031184 podStartE2EDuration="20.953310605s" totalImagesPullingTime="17.755279421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:53.953256264 +0000 UTC m=+21.770021843" watchObservedRunningTime="2026-09-16 18:39:53.953310605 +0000 UTC m=+21.770076190"
Sep 16 18:39:53.953695 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.953674    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bvjjb" podStartSLOduration=3.199107464 podStartE2EDuration="20.953668498s" totalImagesPullingTime="17.754561034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:53.917443693 +0000 UTC m=+21.734209278" watchObservedRunningTime="2026-09-16 18:39:53.953668498 +0000 UTC m=+21.770434086"
Sep 16 18:39:53.982284 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:53.982242    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ph95x" podStartSLOduration=3.223229773 podStartE2EDuration="20.982227649s" totalImagesPullingTime="17.758997876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:53.981128828 +0000 UTC m=+21.797894405" watchObservedRunningTime="2026-09-16 18:39:53.982227649 +0000 UTC m=+21.798993234"
Sep 16 18:39:54.615146 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:54.615107    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:54.615409 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:54.615228    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:54.615409 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:54.615238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:54.615409 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:54.615331    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:54.808348 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:54.808125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" event={"ID":"c0a6e8ff-376d-457f-83d7-b1eb38ccb726","Type":"ContainerStarted","Data":"b928da217b615225a526c3376664c243a561d3895ee29b8948ac5632e115b9d1"}
Sep 16 18:39:54.808348 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:54.808226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" event={"ID":"c0a6e8ff-376d-457f-83d7-b1eb38ccb726","Type":"ContainerStarted","Data":"87d75fd0262507ecb034fc2f84abc6a605decfb2699e5a727f9fc9c7a2cd5e4e"}
Sep 16 18:39:54.808348 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:54.808285    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" event={"ID":"c0a6e8ff-376d-457f-83d7-b1eb38ccb726","Type":"ContainerStarted","Data":"383786b3c1efb190cfadfcd28af825742b17533caed7d02d5b2005f482f5c160"}
Sep 16 18:39:54.929682 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:54.929519    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:39:55.047278 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:55.047255    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:55.576053 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:55.575874    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:39:54.929676671Z","UUID":"7c692908-c22d-4ada-851c-52d38eb85454","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:39:55.578508 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:55.578483    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:39:55.578677 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:55.578521    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:39:55.614185 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:55.614158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:55.614373 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:55.614282    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:55.813019 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:55.812981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xmmnv" event={"ID":"b386d3e2-ddd7-48d2-9c80-3bcf8004ab82","Type":"ContainerStarted","Data":"15f43dde220c4010c4cf64f9f3529a106fdf69c47faf5576cdfcc1185d69a05b"}
Sep 16 18:39:55.815420 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:55.815377    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk" event={"ID":"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645","Type":"ContainerStarted","Data":"f2e7f0e1f4975b5d20033d00910899027e52733503681f5f94f05ffde4c94739"}
Sep 16 18:39:55.832749 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:55.832623    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xmmnv" podStartSLOduration=5.076929074 podStartE2EDuration="22.832604612s" totalImagesPullingTime="17.755675538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:55.832390851 +0000 UTC m=+23.649156437" watchObservedRunningTime="2026-09-16 18:39:55.832604612 +0000 UTC m=+23.649370286"
Sep 16 18:39:56.614704 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:56.614667    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:56.614896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:56.614731    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:56.614896 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:56.614837    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:56.615004 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:56.614960    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:56.819912 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:56.819881    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk" event={"ID":"c53f01d5-60d7-41d0-8d92-c4bd5dcd1645","Type":"ContainerStarted","Data":"d69ed540e2668ed10a5df0f7c9021edd014822a4f710b23475b3aa987b0e5b37"}
Sep 16 18:39:56.823762 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:56.823736    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" event={"ID":"c0a6e8ff-376d-457f-83d7-b1eb38ccb726","Type":"ContainerStarted","Data":"045858c9978169602b6d5cd5b1972a417a4718b65671a49f050c5eec77470b3a"}
Sep 16 18:39:57.614272 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:57.614241    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:57.614431 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:57.614343    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:58.186383 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.186355    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:58.186936 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.186920    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:58.208776 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.208741    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5slk" podStartSLOduration=5.075624122 podStartE2EDuration="25.208705211s" totalImagesPullingTime="20.133081089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:56.844418396 +0000 UTC m=+24.661183982" watchObservedRunningTime="2026-09-16 18:39:58.208705211 +0000 UTC m=+26.025470796"
Sep 16 18:39:58.614517 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.614352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:39:58.614688 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.614352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:39:58.614688 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:58.614583    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:39:58.614688 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:58.614667    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:39:58.830965 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.830891    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" event={"ID":"c0a6e8ff-376d-457f-83d7-b1eb38ccb726","Type":"ContainerStarted","Data":"f193f104f46ea49bc665366fc0b3a005ea0b5e371a3dc9a605a46239488d07c9"}
Sep 16 18:39:58.831205 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.831149    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:58.833008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.832989    2575 generic.go:356] "Generic (PLEG): container finished" podID="6be02496-7563-49b9-90a6-adcce7c4037e" containerID="be48b31f3a698866c2be4813bacce1cb95fa39e2f4beefd34951eb96928e42c7" exitCode=0
Sep 16 18:39:58.833107 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.833069    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgmrl" event={"ID":"6be02496-7563-49b9-90a6-adcce7c4037e","Type":"ContainerDied","Data":"be48b31f3a698866c2be4813bacce1cb95fa39e2f4beefd34951eb96928e42c7"}
Sep 16 18:39:58.833647 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.833619    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nw2jb"
Sep 16 18:39:58.846001 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.845975    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:58.868924 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:58.868884    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd" podStartSLOduration=8.055555116 podStartE2EDuration="25.868873575s" totalImagesPullingTime="17.813318459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:58.868770888 +0000 UTC m=+26.685536473" watchObservedRunningTime="2026-09-16 18:39:58.868873575 +0000 UTC m=+26.685639162"
Sep 16 18:39:59.191614 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:59.191589    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:59.614886 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:59.614855    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:39:59.615051 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:39:59.614963    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:39:59.837210 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:59.837117    2575 generic.go:356] "Generic (PLEG): container finished" podID="6be02496-7563-49b9-90a6-adcce7c4037e" containerID="d76cb8126e969c34df1a12d9dd330373bdef395715e96eee357aa7d840e19b25" exitCode=0
Sep 16 18:39:59.837358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:59.837195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgmrl" event={"ID":"6be02496-7563-49b9-90a6-adcce7c4037e","Type":"ContainerDied","Data":"d76cb8126e969c34df1a12d9dd330373bdef395715e96eee357aa7d840e19b25"}
Sep 16 18:39:59.837795 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:59.837773    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:39:59.854236 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:39:59.854212    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:40:00.515544 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:00.515509    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jjs7q"]
Sep 16 18:40:00.516307 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:00.515630    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:40:00.516307 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:00.515722    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:40:00.516307 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:00.516121    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bd2q"]
Sep 16 18:40:00.516307 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:00.516213    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:40:00.516512 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:00.516306    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:40:00.517172 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:00.517149    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7jzmn"]
Sep 16 18:40:00.517291 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:00.517236    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:40:00.517355 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:00.517337    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:40:00.841468 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:00.841439    2575 generic.go:356] "Generic (PLEG): container finished" podID="6be02496-7563-49b9-90a6-adcce7c4037e" containerID="dea0dad187fb1459a53e83e70136080eff19f43106506eb301d98ba107b5dedd" exitCode=0
Sep 16 18:40:00.841645 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:00.841519    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgmrl" event={"ID":"6be02496-7563-49b9-90a6-adcce7c4037e","Type":"ContainerDied","Data":"dea0dad187fb1459a53e83e70136080eff19f43106506eb301d98ba107b5dedd"}
Sep 16 18:40:02.615617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:02.615422    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:40:02.616068 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:02.615488    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:40:02.616068 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:02.615701    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:40:02.616068 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:02.615521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:40:02.616068 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:02.615769    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:40:02.616068 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:02.615863    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:40:04.614456 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:04.614422    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:40:04.614914 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:04.614422    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:40:04.614914 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:04.614554    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjs7q" podUID="fd64fe4d-7ba3-4355-84b8-8df3b98703c4"
Sep 16 18:40:04.614914 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:04.614422    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:40:04.614914 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:04.614639    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bd2q" podUID="c0e375bf-00f2-427d-a9e8-00b934574883"
Sep 16 18:40:04.614914 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:04.614758    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:40:05.856401 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:05.856364    2575 generic.go:356] "Generic (PLEG): container finished" podID="6be02496-7563-49b9-90a6-adcce7c4037e" containerID="6258ea96b0bdfe40594a2349081bcb1ee09253c0956c281dcc66cd1b2e143976" exitCode=0
Sep 16 18:40:05.856877 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:05.856430    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgmrl" event={"ID":"6be02496-7563-49b9-90a6-adcce7c4037e","Type":"ContainerDied","Data":"6258ea96b0bdfe40594a2349081bcb1ee09253c0956c281dcc66cd1b2e143976"}
Sep 16 18:40:06.280812 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.280725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:40:06.280952 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.280843    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:06.280952 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.280902    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret podName:fd64fe4d-7ba3-4355-84b8-8df3b98703c4 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.280887081 +0000 UTC m=+66.097652645 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret") pod "global-pull-secret-syncer-jjs7q" (UID: "fd64fe4d-7ba3-4355-84b8-8df3b98703c4") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:06.381461 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.381427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:40:06.381461 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.381468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:40:06.381671 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.381562    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:06.381671 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.381584    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:06.381671 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.381604    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:06.381671 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.381613    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kr2tz for pod openshift-network-diagnostics/network-check-target-9bd2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:06.381671 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.381613    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs podName:dca28d1e-3d17-4eea-a864-5f984527ae0f nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.381600174 +0000 UTC m=+66.198365738 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs") pod "network-metrics-daemon-7jzmn" (UID: "dca28d1e-3d17-4eea-a864-5f984527ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:06.381671 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.381663    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz podName:c0e375bf-00f2-427d-a9e8-00b934574883 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.381649042 +0000 UTC m=+66.198414606 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr2tz" (UniqueName: "kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz") pod "network-check-target-9bd2q" (UID: "c0e375bf-00f2-427d-a9e8-00b934574883") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:06.407206 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.407182    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-234.ec2.internal" event="NodeReady"
Sep 16 18:40:06.407344 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.407298    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:40:06.486635 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.486604    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hb7dq"]
Sep 16 18:40:06.510591 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.510565    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jvtwj"]
Sep 16 18:40:06.510740 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.510678    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hb7dq"
Sep 16 18:40:06.517238 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.517218    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:40:06.521228 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.521212    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:40:06.521563 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.521550    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:40:06.530889 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.530843    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7zs8w\""
Sep 16 18:40:06.530943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.530906    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.538860 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.538843    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:40:06.539878 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.539859    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:40:06.539975 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.539961    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-24cbm\""
Sep 16 18:40:06.541169 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.541147    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hb7dq"]
Sep 16 18:40:06.563171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.563146    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jvtwj"]
Sep 16 18:40:06.582406 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.582376    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bddad80-08a6-44db-9fa4-72fb049e7307-cert\") pod \"ingress-canary-hb7dq\" (UID: \"9bddad80-08a6-44db-9fa4-72fb049e7307\") " pod="openshift-ingress-canary/ingress-canary-hb7dq"
Sep 16 18:40:06.582406 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.582408    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8cq\" (UniqueName: \"kubernetes.io/projected/9bddad80-08a6-44db-9fa4-72fb049e7307-kube-api-access-5x8cq\") pod \"ingress-canary-hb7dq\" (UID: \"9bddad80-08a6-44db-9fa4-72fb049e7307\") " pod="openshift-ingress-canary/ingress-canary-hb7dq"
Sep 16 18:40:06.614199 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.614159    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:40:06.614345 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.614165    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:40:06.614345 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.614278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:40:06.625243 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.625221    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:06.626027 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.626005    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:06.626150 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.626045    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rmdqb\""
Sep 16 18:40:06.626212 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.626196    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:06.626801 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.626781    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:06.627235 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.627215    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l6kxj\""
Sep 16 18:40:06.683266 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.683231    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be3ea3a-8122-408d-b74f-54c15a936f07-tmp-dir\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.683266 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.683267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bddad80-08a6-44db-9fa4-72fb049e7307-cert\") pod \"ingress-canary-hb7dq\" (UID: \"9bddad80-08a6-44db-9fa4-72fb049e7307\") " pod="openshift-ingress-canary/ingress-canary-hb7dq"
Sep 16 18:40:06.683455 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.683286    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8cq\" (UniqueName: \"kubernetes.io/projected/9bddad80-08a6-44db-9fa4-72fb049e7307-kube-api-access-5x8cq\") pod \"ingress-canary-hb7dq\" (UID: \"9bddad80-08a6-44db-9fa4-72fb049e7307\") " pod="openshift-ingress-canary/ingress-canary-hb7dq"
Sep 16 18:40:06.683455 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.683319    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.683455 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.683358    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2vr\" (UniqueName: \"kubernetes.io/projected/7be3ea3a-8122-408d-b74f-54c15a936f07-kube-api-access-cl2vr\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.683455 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.683431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7be3ea3a-8122-408d-b74f-54c15a936f07-config-volume\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.687509 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.687488    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bddad80-08a6-44db-9fa4-72fb049e7307-cert\") pod \"ingress-canary-hb7dq\" (UID: \"9bddad80-08a6-44db-9fa4-72fb049e7307\") " pod="openshift-ingress-canary/ingress-canary-hb7dq"
Sep 16 18:40:06.694040 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.694009    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8cq\" (UniqueName: \"kubernetes.io/projected/9bddad80-08a6-44db-9fa4-72fb049e7307-kube-api-access-5x8cq\") pod \"ingress-canary-hb7dq\" (UID: \"9bddad80-08a6-44db-9fa4-72fb049e7307\") " pod="openshift-ingress-canary/ingress-canary-hb7dq"
Sep 16 18:40:06.784277 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.784193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7be3ea3a-8122-408d-b74f-54c15a936f07-config-volume\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.784277 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.784239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be3ea3a-8122-408d-b74f-54c15a936f07-tmp-dir\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.784487 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.784299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.784487 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.784334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2vr\" (UniqueName: \"kubernetes.io/projected/7be3ea3a-8122-408d-b74f-54c15a936f07-kube-api-access-cl2vr\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.784487 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.784443    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:06.784619 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:06.784523    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls podName:7be3ea3a-8122-408d-b74f-54c15a936f07 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:07.284502359 +0000 UTC m=+35.101267932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls") pod "dns-default-jvtwj" (UID: "7be3ea3a-8122-408d-b74f-54c15a936f07") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:06.784619 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.784592    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be3ea3a-8122-408d-b74f-54c15a936f07-tmp-dir\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.784812 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.784783    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7be3ea3a-8122-408d-b74f-54c15a936f07-config-volume\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.799886 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.799865    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2vr\" (UniqueName: \"kubernetes.io/projected/7be3ea3a-8122-408d-b74f-54c15a936f07-kube-api-access-cl2vr\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:06.818813 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.818791    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hb7dq"
Sep 16 18:40:06.862931 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.862767    2575 generic.go:356] "Generic (PLEG): container finished" podID="6be02496-7563-49b9-90a6-adcce7c4037e" containerID="8d58dc184d058bee8f3feae1b4caa672eee8c27aa416aa99b9e5049ec56fc664" exitCode=0
Sep 16 18:40:06.863287 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:06.862821    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgmrl" event={"ID":"6be02496-7563-49b9-90a6-adcce7c4037e","Type":"ContainerDied","Data":"8d58dc184d058bee8f3feae1b4caa672eee8c27aa416aa99b9e5049ec56fc664"}
Sep 16 18:40:07.145256 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:07.145224    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hb7dq"]
Sep 16 18:40:07.149431 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:40:07.149409    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bddad80_08a6_44db_9fa4_72fb049e7307.slice/crio-bd20ed90fda2491587bdaf00f741f3fd754e06d99d03220ff6e9c6b890d3af36 WatchSource:0}: Error finding container bd20ed90fda2491587bdaf00f741f3fd754e06d99d03220ff6e9c6b890d3af36: Status 404 returned error can't find the container with id bd20ed90fda2491587bdaf00f741f3fd754e06d99d03220ff6e9c6b890d3af36
Sep 16 18:40:07.288841 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:07.288798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:07.289003 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:07.288952    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:07.289053 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:07.289019    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls podName:7be3ea3a-8122-408d-b74f-54c15a936f07 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:08.289004403 +0000 UTC m=+36.105769967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls") pod "dns-default-jvtwj" (UID: "7be3ea3a-8122-408d-b74f-54c15a936f07") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:07.869231 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:07.869182    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fgmrl" event={"ID":"6be02496-7563-49b9-90a6-adcce7c4037e","Type":"ContainerStarted","Data":"9426ae81bd83ef4515de77f0bdf085ce3bc7bdd3a61ee1696cf829d0e890c8ac"}
Sep 16 18:40:07.870467 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:07.870438    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hb7dq" event={"ID":"9bddad80-08a6-44db-9fa4-72fb049e7307","Type":"ContainerStarted","Data":"bd20ed90fda2491587bdaf00f741f3fd754e06d99d03220ff6e9c6b890d3af36"}
Sep 16 18:40:08.297588 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:08.297511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:08.297765 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:08.297627    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:08.297765 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:08.297695    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls podName:7be3ea3a-8122-408d-b74f-54c15a936f07 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:10.297677685 +0000 UTC m=+38.114443252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls") pod "dns-default-jvtwj" (UID: "7be3ea3a-8122-408d-b74f-54c15a936f07") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:09.877613 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:09.877372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hb7dq" event={"ID":"9bddad80-08a6-44db-9fa4-72fb049e7307","Type":"ContainerStarted","Data":"480a0afcf09a4eba2b5f8928a6fdaafdd17f5303866b5a073198e59b8552c6f7"}
Sep 16 18:40:09.895945 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:09.895903    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hb7dq" podStartSLOduration=1.442104202 podStartE2EDuration="3.895887358s" totalImagesPullingTime="2.453783156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:09.895098914 +0000 UTC m=+37.711864499" watchObservedRunningTime="2026-09-16 18:40:09.895887358 +0000 UTC m=+37.712652943"
Sep 16 18:40:09.896123 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:09.896105    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fgmrl" podStartSLOduration=9.138678942 podStartE2EDuration="36.896099727s" totalImagesPullingTime="27.757420785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:07.896058775 +0000 UTC m=+35.712824362" watchObservedRunningTime="2026-09-16 18:40:09.896099727 +0000 UTC m=+37.712865313"
Sep 16 18:40:10.311955 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:10.311859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:10.312095 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:10.312031    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:10.312134 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:10.312097    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls podName:7be3ea3a-8122-408d-b74f-54c15a936f07 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:14.312082245 +0000 UTC m=+42.128847826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls") pod "dns-default-jvtwj" (UID: "7be3ea3a-8122-408d-b74f-54c15a936f07") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:14.336006 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:14.335967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:14.336422 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:14.336122    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:14.336422 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:14.336190    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls podName:7be3ea3a-8122-408d-b74f-54c15a936f07 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.33617351 +0000 UTC m=+50.152939075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls") pod "dns-default-jvtwj" (UID: "7be3ea3a-8122-408d-b74f-54c15a936f07") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:22.394984 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:22.394940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:22.395406 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:22.395059    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:22.395406 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:22.395116    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls podName:7be3ea3a-8122-408d-b74f-54c15a936f07 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.395101918 +0000 UTC m=+66.211867500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls") pod "dns-default-jvtwj" (UID: "7be3ea3a-8122-408d-b74f-54c15a936f07") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:31.854286 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:31.854259    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw2kd"
Sep 16 18:40:38.301446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.301404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:40:38.304777 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.304756    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:38.315329 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.315307    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd64fe4d-7ba3-4355-84b8-8df3b98703c4-original-pull-secret\") pod \"global-pull-secret-syncer-jjs7q\" (UID: \"fd64fe4d-7ba3-4355-84b8-8df3b98703c4\") " pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:40:38.402047 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.402010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:40:38.402047 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.402051    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:40:38.402209 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.402088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:40:38.402257 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:38.402208    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:38.402295 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:38.402272    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls podName:7be3ea3a-8122-408d-b74f-54c15a936f07 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:10.402258542 +0000 UTC m=+98.219024106 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls") pod "dns-default-jvtwj" (UID: "7be3ea3a-8122-408d-b74f-54c15a936f07") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:38.405113 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.405098    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:38.405420 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.405404    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:38.412527 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:38.412506    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:40:38.412661 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:40:38.412649    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs podName:dca28d1e-3d17-4eea-a864-5f984527ae0f nodeName:}" failed. No retries permitted until 2026-09-16 18:41:42.412634568 +0000 UTC m=+130.229400137 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs") pod "network-metrics-daemon-7jzmn" (UID: "dca28d1e-3d17-4eea-a864-5f984527ae0f") : secret "metrics-daemon-secret" not found
Sep 16 18:40:38.416799 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.416783    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:38.425690 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.425662    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2tz\" (UniqueName: \"kubernetes.io/projected/c0e375bf-00f2-427d-a9e8-00b934574883-kube-api-access-kr2tz\") pod \"network-check-target-9bd2q\" (UID: \"c0e375bf-00f2-427d-a9e8-00b934574883\") " pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:40:38.432465 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.432447    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjs7q"
Sep 16 18:40:38.711277 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.711247    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jjs7q"]
Sep 16 18:40:38.714901 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:40:38.714875    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd64fe4d_7ba3_4355_84b8_8df3b98703c4.slice/crio-1c0239a954f1c4d65407223c88f31b8872b4f68562eb0b6abd2f09f022e02725 WatchSource:0}: Error finding container 1c0239a954f1c4d65407223c88f31b8872b4f68562eb0b6abd2f09f022e02725: Status 404 returned error can't find the container with id 1c0239a954f1c4d65407223c88f31b8872b4f68562eb0b6abd2f09f022e02725
Sep 16 18:40:38.725763 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.725742    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rmdqb\""
Sep 16 18:40:38.733428 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.733409    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:40:38.943594 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:38.943555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jjs7q" event={"ID":"fd64fe4d-7ba3-4355-84b8-8df3b98703c4","Type":"ContainerStarted","Data":"1c0239a954f1c4d65407223c88f31b8872b4f68562eb0b6abd2f09f022e02725"}
Sep 16 18:40:39.023006 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:39.022943    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bd2q"]
Sep 16 18:40:39.026334 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:40:39.026306    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e375bf_00f2_427d_a9e8_00b934574883.slice/crio-42d11be6f9414ec9ed9c73d792c4f618850bc7747d2e68489a133e773beba7be WatchSource:0}: Error finding container 42d11be6f9414ec9ed9c73d792c4f618850bc7747d2e68489a133e773beba7be: Status 404 returned error can't find the container with id 42d11be6f9414ec9ed9c73d792c4f618850bc7747d2e68489a133e773beba7be
Sep 16 18:40:39.948264 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:39.948224    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bd2q" event={"ID":"c0e375bf-00f2-427d-a9e8-00b934574883","Type":"ContainerStarted","Data":"42d11be6f9414ec9ed9c73d792c4f618850bc7747d2e68489a133e773beba7be"}
Sep 16 18:40:43.959170 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:43.959130    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jjs7q" event={"ID":"fd64fe4d-7ba3-4355-84b8-8df3b98703c4","Type":"ContainerStarted","Data":"c33c600fe3b37b1e0dc69d00397c2d12986ebcba6c8486eeff6bcf1477df9f2e"}
Sep 16 18:40:43.960754 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:43.960726    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bd2q" event={"ID":"c0e375bf-00f2-427d-a9e8-00b934574883","Type":"ContainerStarted","Data":"4959293b3c1bc591bac46b4233e88b29e4488f26392b1569838ac409711266ab"}
Sep 16 18:40:43.960874 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:43.960858    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:40:44.005749 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:40:44.005693    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jjs7q" podStartSLOduration=66.464713664 podStartE2EDuration="1m11.005679715s" totalImagesPullingTime="4.540966051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:43.983388267 +0000 UTC m=+71.800153853" watchObservedRunningTime="2026-09-16 18:40:44.005679715 +0000 UTC m=+71.822445295"
Sep 16 18:41:10.415913 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:10.415878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:41:10.416256 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:10.416029    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:10.416256 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:10.416089    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls podName:7be3ea3a-8122-408d-b74f-54c15a936f07 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.416074686 +0000 UTC m=+162.232840255 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls") pod "dns-default-jvtwj" (UID: "7be3ea3a-8122-408d-b74f-54c15a936f07") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:14.964887 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:14.964854    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9bd2q"
Sep 16 18:41:15.025616 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:15.025568    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9bd2q" podStartSLOduration=97.802864432 podStartE2EDuration="1m42.025552818s" totalImagesPullingTime="4.222688386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:44.006608673 +0000 UTC m=+71.823374261" watchObservedRunningTime="2026-09-16 18:41:15.025552818 +0000 UTC m=+102.842318403"
Sep 16 18:41:31.974123 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:31.974090    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"]
Sep 16 18:41:31.975875 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:31.975857    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:31.998471 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:31.998442    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:41:31.998471 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:31.998462    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:31.999178 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:31.999163    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:31.999229 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:31.999178    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xkjgx\""
Sep 16 18:41:31.999229 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:31.999162    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:41:32.058504 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.058484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ecea926-c56c-4436-9403-3484377dbfed-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-plmd7\" (UID: \"5ecea926-c56c-4436-9403-3484377dbfed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.058620 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.058528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ecea926-c56c-4436-9403-3484377dbfed-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-plmd7\" (UID: \"5ecea926-c56c-4436-9403-3484377dbfed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.058620 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.058544    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvr9n\" (UniqueName: \"kubernetes.io/projected/5ecea926-c56c-4436-9403-3484377dbfed-kube-api-access-tvr9n\") pod \"kube-storage-version-migrator-operator-6c8f84f897-plmd7\" (UID: \"5ecea926-c56c-4436-9403-3484377dbfed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.157953 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.157931    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"]
Sep 16 18:41:32.159302 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.159277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ecea926-c56c-4436-9403-3484377dbfed-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-plmd7\" (UID: \"5ecea926-c56c-4436-9403-3484377dbfed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.159416 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.159324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ecea926-c56c-4436-9403-3484377dbfed-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-plmd7\" (UID: \"5ecea926-c56c-4436-9403-3484377dbfed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.159416 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.159344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvr9n\" (UniqueName: \"kubernetes.io/projected/5ecea926-c56c-4436-9403-3484377dbfed-kube-api-access-tvr9n\") pod \"kube-storage-version-migrator-operator-6c8f84f897-plmd7\" (UID: \"5ecea926-c56c-4436-9403-3484377dbfed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.159895 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.159873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ecea926-c56c-4436-9403-3484377dbfed-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-plmd7\" (UID: \"5ecea926-c56c-4436-9403-3484377dbfed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.161906 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.161890    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ecea926-c56c-4436-9403-3484377dbfed-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-plmd7\" (UID: \"5ecea926-c56c-4436-9403-3484377dbfed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.184075 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.184044    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvr9n\" (UniqueName: \"kubernetes.io/projected/5ecea926-c56c-4436-9403-3484377dbfed-kube-api-access-tvr9n\") pod \"kube-storage-version-migrator-operator-6c8f84f897-plmd7\" (UID: \"5ecea926-c56c-4436-9403-3484377dbfed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.256850 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.256791    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c4c9fb988-ncv47"]
Sep 16 18:41:32.258803 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.258782    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.262544 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.262528    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:41:32.265699 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.265680    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:41:32.266490 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.266473    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:41:32.270764 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.270748    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:41:32.271158 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.271146    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x6nbc\""
Sep 16 18:41:32.285483 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.285464    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:41:32.286514 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.286501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"
Sep 16 18:41:32.287940 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.287923    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:41:32.333319 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.333252    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c4c9fb988-ncv47"]
Sep 16 18:41:32.360256 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.360234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.360372 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.360266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.360372 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.360289    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-default-certificate\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.360372 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.360312    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-stats-auth\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.360372 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.360333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsqr\" (UniqueName: \"kubernetes.io/projected/0673ac02-7d60-466c-9f91-bc2f1b19eafb-kube-api-access-jnsqr\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.453661 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.453632    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"]
Sep 16 18:41:32.455815 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.455797    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.456546 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.456527    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5998b9f4f4-45jkq"]
Sep 16 18:41:32.458191 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.458174    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.460470 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.460449    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:32.460549 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.460459    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:41:32.460594 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.460541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.460594 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.460579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-default-certificate\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.460691 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.460617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-stats-auth\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.460691 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:32.460662    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:32.960644422 +0000 UTC m=+120.777410002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:32.460691 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.460681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsqr\" (UniqueName: \"kubernetes.io/projected/0673ac02-7d60-466c-9f91-bc2f1b19eafb-kube-api-access-jnsqr\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.460898 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.460763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.460898 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:32.460848    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:32.460898 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:32.460891    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:32.960876986 +0000 UTC m=+120.777642554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : secret "router-metrics-certs-default" not found
Sep 16 18:41:32.461099 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.460995    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:41:32.461171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.461141    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xj678\""
Sep 16 18:41:32.461223 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.461190    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:32.462064 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.462046    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:41:32.462163 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.462050    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:41:32.462546 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.462523    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:41:32.463281 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.463257    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-stats-auth\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.463349 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.463313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-default-certificate\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.472385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.472367    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tm7mf\""
Sep 16 18:41:32.474101 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.474087    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:41:32.510202 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.510158    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:41:32.514102 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.514085    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5998b9f4f4-45jkq"]
Sep 16 18:41:32.518665 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.518648    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:41:32.528082 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.528064    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsqr\" (UniqueName: \"kubernetes.io/projected/0673ac02-7d60-466c-9f91-bc2f1b19eafb-kube-api-access-jnsqr\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.561159 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561136    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-trusted-ca\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.561234 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxdz\" (UniqueName: \"kubernetes.io/projected/fd80257a-afc9-48f9-b887-6053066bafbf-kube-api-access-frxdz\") pod \"service-ca-operator-cf7c776b7-zrcbw\" (UID: \"fd80257a-afc9-48f9-b887-6053066bafbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.561234 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561194    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd80257a-afc9-48f9-b887-6053066bafbf-serving-cert\") pod \"service-ca-operator-cf7c776b7-zrcbw\" (UID: \"fd80257a-afc9-48f9-b887-6053066bafbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.561234 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79199211-4552-4acb-945b-56728721546f-ca-trust-extracted\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.561234 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-bound-sa-token\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.561358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd80257a-afc9-48f9-b887-6053066bafbf-config\") pod \"service-ca-operator-cf7c776b7-zrcbw\" (UID: \"fd80257a-afc9-48f9-b887-6053066bafbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.561358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.561358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-installation-pull-secrets\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.561449 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfr85\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-kube-api-access-bfr85\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.561449 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561408    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-image-registry-private-configuration\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.561509 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.561443    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-registry-certificates\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.567193 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.567173    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"]
Sep 16 18:41:32.631224 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.631203    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7"]
Sep 16 18:41:32.635125 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:41:32.635106    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecea926_c56c_4436_9403_3484377dbfed.slice/crio-7bb48f184e8f33f0f1714422eb18b1da9e442c7de5bb13ece216839c6b59a773 WatchSource:0}: Error finding container 7bb48f184e8f33f0f1714422eb18b1da9e442c7de5bb13ece216839c6b59a773: Status 404 returned error can't find the container with id 7bb48f184e8f33f0f1714422eb18b1da9e442c7de5bb13ece216839c6b59a773
Sep 16 18:41:32.662155 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxdz\" (UniqueName: \"kubernetes.io/projected/fd80257a-afc9-48f9-b887-6053066bafbf-kube-api-access-frxdz\") pod \"service-ca-operator-cf7c776b7-zrcbw\" (UID: \"fd80257a-afc9-48f9-b887-6053066bafbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.662252 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd80257a-afc9-48f9-b887-6053066bafbf-serving-cert\") pod \"service-ca-operator-cf7c776b7-zrcbw\" (UID: \"fd80257a-afc9-48f9-b887-6053066bafbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.662252 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79199211-4552-4acb-945b-56728721546f-ca-trust-extracted\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.662252 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-bound-sa-token\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.662252 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd80257a-afc9-48f9-b887-6053066bafbf-config\") pod \"service-ca-operator-cf7c776b7-zrcbw\" (UID: \"fd80257a-afc9-48f9-b887-6053066bafbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.662252 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.662507 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-installation-pull-secrets\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.662507 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662333    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfr85\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-kube-api-access-bfr85\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.662507 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-image-registry-private-configuration\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.662507 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-registry-certificates\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.662507 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-trusted-ca\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.662748 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662600    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79199211-4552-4acb-945b-56728721546f-ca-trust-extracted\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.662942 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.662922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-registry-certificates\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.693946 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.693924    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:41:32.694166 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.694151    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:41:32.694220 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.694177    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:41:32.694275 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.694237    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:41:32.701669 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.701653    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:41:32.702557 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:32.702538    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:32.702625 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:32.702560    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5998b9f4f4-45jkq: secret "image-registry-tls" not found
Sep 16 18:41:32.702625 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:32.702611    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls podName:79199211-4552-4acb-945b-56728721546f nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.202593738 +0000 UTC m=+121.019359302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls") pod "image-registry-5998b9f4f4-45jkq" (UID: "79199211-4552-4acb-945b-56728721546f") : secret "image-registry-tls" not found
Sep 16 18:41:32.702881 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.702847    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-bound-sa-token\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.702988 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.702974    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd80257a-afc9-48f9-b887-6053066bafbf-config\") pod \"service-ca-operator-cf7c776b7-zrcbw\" (UID: \"fd80257a-afc9-48f9-b887-6053066bafbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.704514 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.704489    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd80257a-afc9-48f9-b887-6053066bafbf-serving-cert\") pod \"service-ca-operator-cf7c776b7-zrcbw\" (UID: \"fd80257a-afc9-48f9-b887-6053066bafbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.704659 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.704641    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-installation-pull-secrets\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.704743 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.704728    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-image-registry-private-configuration\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.713328 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.713315    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:32.720503 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.720488    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:32.727417 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.727396    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxdz\" (UniqueName: \"kubernetes.io/projected/fd80257a-afc9-48f9-b887-6053066bafbf-kube-api-access-frxdz\") pod \"service-ca-operator-cf7c776b7-zrcbw\" (UID: \"fd80257a-afc9-48f9-b887-6053066bafbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.727795 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.727777    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:41:32.729328 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.729308    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfr85\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-kube-api-access-bfr85\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.733622 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.733606    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-trusted-ca\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:32.775224 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.775161    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xj678\""
Sep 16 18:41:32.781979 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.781963    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"
Sep 16 18:41:32.964884 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.964833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.965061 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:32.964895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:32.965061 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:32.964982    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:32.965061 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:32.965004    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.964988679 +0000 UTC m=+121.781754257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:32.965061 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:32.965033    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.965020926 +0000 UTC m=+121.781786495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : secret "router-metrics-certs-default" not found
Sep 16 18:41:33.063940 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:33.063913    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw"]
Sep 16 18:41:33.065243 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:41:33.065219    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd80257a_afc9_48f9_b887_6053066bafbf.slice/crio-27f7c690a2f076c15d2218787935c0fa3a2acfb2fe6109ddb0c6da165edb4b32 WatchSource:0}: Error finding container 27f7c690a2f076c15d2218787935c0fa3a2acfb2fe6109ddb0c6da165edb4b32: Status 404 returned error can't find the container with id 27f7c690a2f076c15d2218787935c0fa3a2acfb2fe6109ddb0c6da165edb4b32
Sep 16 18:41:33.078129 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:33.078100    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7" event={"ID":"5ecea926-c56c-4436-9403-3484377dbfed","Type":"ContainerStarted","Data":"7bb48f184e8f33f0f1714422eb18b1da9e442c7de5bb13ece216839c6b59a773"}
Sep 16 18:41:33.079077 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:33.079056    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw" event={"ID":"fd80257a-afc9-48f9-b887-6053066bafbf","Type":"ContainerStarted","Data":"27f7c690a2f076c15d2218787935c0fa3a2acfb2fe6109ddb0c6da165edb4b32"}
Sep 16 18:41:33.266865 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:33.266831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:33.267028 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:33.267006    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:33.267086 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:33.267034    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5998b9f4f4-45jkq: secret "image-registry-tls" not found
Sep 16 18:41:33.267123 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:33.267099    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls podName:79199211-4552-4acb-945b-56728721546f nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.267080935 +0000 UTC m=+122.083846500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls") pod "image-registry-5998b9f4f4-45jkq" (UID: "79199211-4552-4acb-945b-56728721546f") : secret "image-registry-tls" not found
Sep 16 18:41:33.972125 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:33.972073    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:33.972285 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:33.972234    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:35.972210866 +0000 UTC m=+123.788976434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:33.972285 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:33.972257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:33.972397 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:33.972382    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:33.972454 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:33.972443    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:35.972424995 +0000 UTC m=+123.789190576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : secret "router-metrics-certs-default" not found
Sep 16 18:41:34.274987 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:34.274905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:34.275427 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:34.275057    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:34.275427 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:34.275073    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5998b9f4f4-45jkq: secret "image-registry-tls" not found
Sep 16 18:41:34.275427 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:34.275131    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls podName:79199211-4552-4acb-945b-56728721546f nodeName:}" failed. No retries permitted until 2026-09-16 18:41:36.275112023 +0000 UTC m=+124.091877592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls") pod "image-registry-5998b9f4f4-45jkq" (UID: "79199211-4552-4acb-945b-56728721546f") : secret "image-registry-tls" not found
Sep 16 18:41:35.085318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:35.085287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7" event={"ID":"5ecea926-c56c-4436-9403-3484377dbfed","Type":"ContainerStarted","Data":"57048607dcc685af9801e7f83bdfb41ae32466440f92bcd081ed90be5bc05ede"}
Sep 16 18:41:35.086870 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:35.086842    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw" event={"ID":"fd80257a-afc9-48f9-b887-6053066bafbf","Type":"ContainerStarted","Data":"32a2e21837b0ed8d0d351af1cc49d3db89fceb9724cfcc0ae9f6d3c6c9fe2b03"}
Sep 16 18:41:35.116444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:35.116393    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7" podStartSLOduration=2.336331111 podStartE2EDuration="4.116372629s" totalImagesPullingTime="1.780041518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:35.115361967 +0000 UTC m=+122.932127552" watchObservedRunningTime="2026-09-16 18:41:35.116372629 +0000 UTC m=+122.933138216"
Sep 16 18:41:35.987222 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:35.987174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:35.987604 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:35.987244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:35.987604 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:35.987315    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:35.987604 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:35.987386    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:39.987371805 +0000 UTC m=+127.804137369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : secret "router-metrics-certs-default" not found
Sep 16 18:41:35.987604 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:35.987416    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:39.987398166 +0000 UTC m=+127.804163737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:36.289226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:36.289134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:36.289360 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:36.289287    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:36.289360 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:36.289305    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5998b9f4f4-45jkq: secret "image-registry-tls" not found
Sep 16 18:41:36.289443 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:36.289361    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls podName:79199211-4552-4acb-945b-56728721546f nodeName:}" failed. No retries permitted until 2026-09-16 18:41:40.289344843 +0000 UTC m=+128.106110427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls") pod "image-registry-5998b9f4f4-45jkq" (UID: "79199211-4552-4acb-945b-56728721546f") : secret "image-registry-tls" not found
Sep 16 18:41:36.922081 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:36.922027    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw" podStartSLOduration=2.957774137 podStartE2EDuration="4.922010157s" totalImagesPullingTime="1.96423602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:35.145382016 +0000 UTC m=+122.962147602" watchObservedRunningTime="2026-09-16 18:41:36.922010157 +0000 UTC m=+124.738775749"
Sep 16 18:41:36.922246 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:36.922210    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm"]
Sep 16 18:41:36.924410 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:36.924379    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm"
Sep 16 18:41:36.929783 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:36.929759    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:41:36.935574 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:36.935544    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:36.935658 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:36.935547    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9dt2s\""
Sep 16 18:41:36.996377 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:36.996351    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljjf\" (UniqueName: \"kubernetes.io/projected/04d89af1-bce5-45c8-a85f-af26c0215d19-kube-api-access-8ljjf\") pod \"migrator-8b4d697c5-pz4lm\" (UID: \"04d89af1-bce5-45c8-a85f-af26c0215d19\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm"
Sep 16 18:41:37.083796 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:37.083758    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm"]
Sep 16 18:41:37.096846 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:37.096820    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljjf\" (UniqueName: \"kubernetes.io/projected/04d89af1-bce5-45c8-a85f-af26c0215d19-kube-api-access-8ljjf\") pod \"migrator-8b4d697c5-pz4lm\" (UID: \"04d89af1-bce5-45c8-a85f-af26c0215d19\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm"
Sep 16 18:41:37.111221 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:37.111192    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljjf\" (UniqueName: \"kubernetes.io/projected/04d89af1-bce5-45c8-a85f-af26c0215d19-kube-api-access-8ljjf\") pod \"migrator-8b4d697c5-pz4lm\" (UID: \"04d89af1-bce5-45c8-a85f-af26c0215d19\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm"
Sep 16 18:41:37.232471 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:37.232381    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm"
Sep 16 18:41:37.507226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:37.507160    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm"]
Sep 16 18:41:37.510348 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:41:37.510320    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d89af1_bce5_45c8_a85f_af26c0215d19.slice/crio-cd014ba613852e1e8ba3a49d59fc56bdce6ec3fb723b869770a02a916b0a148d WatchSource:0}: Error finding container cd014ba613852e1e8ba3a49d59fc56bdce6ec3fb723b869770a02a916b0a148d: Status 404 returned error can't find the container with id cd014ba613852e1e8ba3a49d59fc56bdce6ec3fb723b869770a02a916b0a148d
Sep 16 18:41:38.100672 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.100634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm" event={"ID":"04d89af1-bce5-45c8-a85f-af26c0215d19","Type":"ContainerStarted","Data":"cd014ba613852e1e8ba3a49d59fc56bdce6ec3fb723b869770a02a916b0a148d"}
Sep 16 18:41:38.867781 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.867752    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-clcb7"]
Sep 16 18:41:38.869685 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.869663    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:38.901088 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.901063    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:41:38.901346 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.901332    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:41:38.904852 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.904835    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-blflj\""
Sep 16 18:41:38.904990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.904974    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:41:38.905382 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.905368    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:41:38.909662 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.909644    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqqqt\" (UniqueName: \"kubernetes.io/projected/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-api-access-xqqqt\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:38.909750 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.909685    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:38.909750 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.909728    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8857acc3-4e6e-4f99-b382-5b9f3c460f20-data-volume\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:38.909837 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.909765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:38.909837 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.909808    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8857acc3-4e6e-4f99-b382-5b9f3c460f20-crio-socket\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:38.941574 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:38.941551    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-clcb7"]
Sep 16 18:41:39.011133 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.011101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8857acc3-4e6e-4f99-b382-5b9f3c460f20-crio-socket\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.011312 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.011250    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8857acc3-4e6e-4f99-b382-5b9f3c460f20-crio-socket\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.011312 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.011266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqqt\" (UniqueName: \"kubernetes.io/projected/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-api-access-xqqqt\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.011431 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.011316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.011431 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.011358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8857acc3-4e6e-4f99-b382-5b9f3c460f20-data-volume\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.011431 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.011383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.011568 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:39.011478    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:39.011568 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:39.011539    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:39.511523505 +0000 UTC m=+127.328289069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:39.011568 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:39.011480    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:39.011697 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:39.011612    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:39.511601075 +0000 UTC m=+127.328366639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:39.011697 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.011661    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8857acc3-4e6e-4f99-b382-5b9f3c460f20-data-volume\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.055768 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.055744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqqt\" (UniqueName: \"kubernetes.io/projected/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-api-access-xqqqt\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.105451 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.105417    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm" event={"ID":"04d89af1-bce5-45c8-a85f-af26c0215d19","Type":"ContainerStarted","Data":"4834af2a7cd31ea59a77d2db1437625e46c8e6c40c363ff9d7da9edce741722d"}
Sep 16 18:41:39.105788 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.105477    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm" event={"ID":"04d89af1-bce5-45c8-a85f-af26c0215d19","Type":"ContainerStarted","Data":"3bbb36ab249119447c5547ad3d04e6e81c72fd3f05b6bd53f53e62363d35ce48"}
Sep 16 18:41:39.515662 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.515624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.515855 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:39.515674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:39.515855 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:39.515786    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:39.515953 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:39.515857    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:40.515841759 +0000 UTC m=+128.332607336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:39.515953 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:39.515794    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:39.515953 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:39.515935    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:40.515922183 +0000 UTC m=+128.332687759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:40.020624 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:40.020573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:40.020624 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:40.020631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:40.020905 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.020735    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:40.020905 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.020805    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:48.020791332 +0000 UTC m=+135.837556896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : secret "router-metrics-certs-default" not found
Sep 16 18:41:40.020905 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.020820    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:41:48.020814061 +0000 UTC m=+135.837579624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:40.323477 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:40.323379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:40.323910 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.323533    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:40.323910 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.323559    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5998b9f4f4-45jkq: secret "image-registry-tls" not found
Sep 16 18:41:40.323910 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.323615    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls podName:79199211-4552-4acb-945b-56728721546f nodeName:}" failed. No retries permitted until 2026-09-16 18:41:48.323599365 +0000 UTC m=+136.140364933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls") pod "image-registry-5998b9f4f4-45jkq" (UID: "79199211-4552-4acb-945b-56728721546f") : secret "image-registry-tls" not found
Sep 16 18:41:40.524945 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:40.524904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:40.524945 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:40.524952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:40.525152 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.525063    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:40.525152 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.525125    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:40.525215 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.525128    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:42.525111338 +0000 UTC m=+130.341876922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:40.525215 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:40.525174    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:42.525163165 +0000 UTC m=+130.341928732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.386719 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:41.386672    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ph95x_e06d59a6-6b82-4c92-adc3-c579ab24cffd/dns-node-resolver/0.log"
Sep 16 18:41:42.440864 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:42.440815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:41:42.441243 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:42.440975    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:42.441243 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:42.441040    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs podName:dca28d1e-3d17-4eea-a864-5f984527ae0f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:44.441026178 +0000 UTC m=+252.257791746 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs") pod "network-metrics-daemon-7jzmn" (UID: "dca28d1e-3d17-4eea-a864-5f984527ae0f") : secret "metrics-daemon-secret" not found
Sep 16 18:41:42.542123 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:42.542074    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:42.542123 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:42.542134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:42.542315 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:42.542205    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:42.542315 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:42.542257    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:46.542243885 +0000 UTC m=+134.359009449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:42.542315 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:42.542253    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:42.542315 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:42.542282    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:46.542276127 +0000 UTC m=+134.359041692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:42.590025 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:42.590002    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bvjjb_ae87ca82-6c33-4412-87f9-92ba9f2858f3/node-ca/0.log"
Sep 16 18:41:43.213573 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:43.213534    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hb7dq_9bddad80-08a6-44db-9fa4-72fb049e7307/serve-healthcheck-canary/0.log"
Sep 16 18:41:43.595455 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:43.595427    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-pz4lm_04d89af1-bce5-45c8-a85f-af26c0215d19/migrator/0.log"
Sep 16 18:41:43.785876 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:43.785850    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-pz4lm_04d89af1-bce5-45c8-a85f-af26c0215d19/graceful-termination/0.log"
Sep 16 18:41:43.996758 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:43.996674    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-plmd7_5ecea926-c56c-4436-9403-3484377dbfed/kube-storage-version-migrator-operator/0.log"
Sep 16 18:41:46.575420 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:46.575367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:46.575420 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:46.575425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:46.575883 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:46.575503    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:46.575883 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:46.575525    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:46.575883 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:46.575558    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:54.575544716 +0000 UTC m=+142.392310280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:46.575883 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:46.575583    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls podName:8857acc3-4e6e-4f99-b382-5b9f3c460f20 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:54.575569758 +0000 UTC m=+142.392335343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clcb7" (UID: "8857acc3-4e6e-4f99-b382-5b9f3c460f20") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:48.087816 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:48.087778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:48.087816 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:48.087817    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:48.088217 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:41:48.087979    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle podName:0673ac02-7d60-466c-9f91-bc2f1b19eafb nodeName:}" failed. No retries permitted until 2026-09-16 18:42:04.087963773 +0000 UTC m=+151.904729336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle") pod "router-default-6c4c9fb988-ncv47" (UID: "0673ac02-7d60-466c-9f91-bc2f1b19eafb") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:48.090315 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:48.090290    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0673ac02-7d60-466c-9f91-bc2f1b19eafb-metrics-certs\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:41:48.390676 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:48.390588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:48.393029 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:48.393003    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") pod \"image-registry-5998b9f4f4-45jkq\" (UID: \"79199211-4552-4acb-945b-56728721546f\") " pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:48.682644 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:48.682574    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tm7mf\""
Sep 16 18:41:48.687822 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:48.687808    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:49.058190 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:49.058097    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-pz4lm" podStartSLOduration=12.208763356 podStartE2EDuration="13.058080187s" totalImagesPullingTime="849.316831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:39.153013419 +0000 UTC m=+126.969779004" watchObservedRunningTime="2026-09-16 18:41:49.058080187 +0000 UTC m=+136.874845772"
Sep 16 18:41:49.058620 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:49.058604    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5998b9f4f4-45jkq"]
Sep 16 18:41:49.061612 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:41:49.061585    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79199211_4552_4acb_945b_56728721546f.slice/crio-d70a45ce48608817c1ef9d3dda1603f70ebfd8bc7cec72251224a40bda641cc1 WatchSource:0}: Error finding container d70a45ce48608817c1ef9d3dda1603f70ebfd8bc7cec72251224a40bda641cc1: Status 404 returned error can't find the container with id d70a45ce48608817c1ef9d3dda1603f70ebfd8bc7cec72251224a40bda641cc1
Sep 16 18:41:49.136945 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:49.136917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq" event={"ID":"79199211-4552-4acb-945b-56728721546f","Type":"ContainerStarted","Data":"99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad"}
Sep 16 18:41:49.137334 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:49.136962    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq" event={"ID":"79199211-4552-4acb-945b-56728721546f","Type":"ContainerStarted","Data":"d70a45ce48608817c1ef9d3dda1603f70ebfd8bc7cec72251224a40bda641cc1"}
Sep 16 18:41:49.137334 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:49.137063    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:41:54.640859 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:54.640827    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:54.641212 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:54.640870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:54.641399 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:54.641380    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8857acc3-4e6e-4f99-b382-5b9f3c460f20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:54.643161 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:54.643143    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8857acc3-4e6e-4f99-b382-5b9f3c460f20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clcb7\" (UID: \"8857acc3-4e6e-4f99-b382-5b9f3c460f20\") " pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:54.777603 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:54.777579    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-clcb7"
Sep 16 18:41:55.056546 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:55.056447    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq" podStartSLOduration=23.056429571 podStartE2EDuration="23.056429571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:49.230188459 +0000 UTC m=+137.046954044" watchObservedRunningTime="2026-09-16 18:41:55.056429571 +0000 UTC m=+142.873195156"
Sep 16 18:41:55.057331 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:55.057304    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-clcb7"]
Sep 16 18:41:55.060346 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:41:55.060319    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8857acc3_4e6e_4f99_b382_5b9f3c460f20.slice/crio-6cb02143705dfebe6811e702c6e96d602550d49cb4d948dc8bb51275c5826cf8 WatchSource:0}: Error finding container 6cb02143705dfebe6811e702c6e96d602550d49cb4d948dc8bb51275c5826cf8: Status 404 returned error can't find the container with id 6cb02143705dfebe6811e702c6e96d602550d49cb4d948dc8bb51275c5826cf8
Sep 16 18:41:55.158828 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:55.158792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clcb7" event={"ID":"8857acc3-4e6e-4f99-b382-5b9f3c460f20","Type":"ContainerStarted","Data":"b3a02095cf17a7a30f53ecbc95cd64bc14a1a550f110fcfd7e25b629283432e1"}
Sep 16 18:41:55.159041 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:55.158859    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clcb7" event={"ID":"8857acc3-4e6e-4f99-b382-5b9f3c460f20","Type":"ContainerStarted","Data":"6cb02143705dfebe6811e702c6e96d602550d49cb4d948dc8bb51275c5826cf8"}
Sep 16 18:41:56.164238 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:56.164198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clcb7" event={"ID":"8857acc3-4e6e-4f99-b382-5b9f3c460f20","Type":"ContainerStarted","Data":"d42b2d44615e42a01ba0c30e101236ab4ad5a939d7d6976605dd154140b9b94f"}
Sep 16 18:41:57.168637 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:57.168602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clcb7" event={"ID":"8857acc3-4e6e-4f99-b382-5b9f3c460f20","Type":"ContainerStarted","Data":"5aea694d787cf5f06572564ebeeb52f3b0d4729b2d5ee6d4d3dec3a47e690a3f"}
Sep 16 18:41:57.203298 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:41:57.203254    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-clcb7" podStartSLOduration=17.571488668 podStartE2EDuration="19.203237934s" totalImagesPullingTime="1.631749266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:57.200626265 +0000 UTC m=+145.017391850" watchObservedRunningTime="2026-09-16 18:41:57.203237934 +0000 UTC m=+145.020003519"
Sep 16 18:42:04.117857 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:04.117814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:42:04.118423 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:04.118401    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0673ac02-7d60-466c-9f91-bc2f1b19eafb-service-ca-bundle\") pod \"router-default-6c4c9fb988-ncv47\" (UID: \"0673ac02-7d60-466c-9f91-bc2f1b19eafb\") " pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:42:04.373271 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:04.373196    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x6nbc\""
Sep 16 18:42:04.377416 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:04.377397    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:42:04.445920 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:04.445896    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5998b9f4f4-45jkq"]
Sep 16 18:42:04.450290 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:04.450238    2575 patch_prober.go:28] interesting pod/image-registry-5998b9f4f4-45jkq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:42:04.450434 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:04.450304    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq" podUID="79199211-4552-4acb-945b-56728721546f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:42:04.741395 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:04.741323    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c4c9fb988-ncv47"]
Sep 16 18:42:04.744287 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:04.744259    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0673ac02_7d60_466c_9f91_bc2f1b19eafb.slice/crio-9c9c9ea1dff0d185020fc7cfaec7eefb88111056f2104dc34e5e1e37fbf80994 WatchSource:0}: Error finding container 9c9c9ea1dff0d185020fc7cfaec7eefb88111056f2104dc34e5e1e37fbf80994: Status 404 returned error can't find the container with id 9c9c9ea1dff0d185020fc7cfaec7eefb88111056f2104dc34e5e1e37fbf80994
Sep 16 18:42:05.194063 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:05.194029    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c4c9fb988-ncv47" event={"ID":"0673ac02-7d60-466c-9f91-bc2f1b19eafb","Type":"ContainerStarted","Data":"ebc096f86f16cb0177ceed0590cd0af727eb1c63bf7ce539a97147bcf0bb962e"}
Sep 16 18:42:05.194419 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:05.194107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c4c9fb988-ncv47" event={"ID":"0673ac02-7d60-466c-9f91-bc2f1b19eafb","Type":"ContainerStarted","Data":"9c9c9ea1dff0d185020fc7cfaec7eefb88111056f2104dc34e5e1e37fbf80994"}
Sep 16 18:42:05.378526 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:05.378486    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:42:05.380894 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:05.380872    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:42:05.466809 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:05.466677    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6c4c9fb988-ncv47" podStartSLOduration=33.466661755 podStartE2EDuration="33.466661755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:05.260601629 +0000 UTC m=+153.077367216" watchObservedRunningTime="2026-09-16 18:42:05.466661755 +0000 UTC m=+153.283427341"
Sep 16 18:42:06.197215 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:06.197183    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:42:06.198428 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:06.198405    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c4c9fb988-ncv47"
Sep 16 18:42:07.297451 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.297414    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"]
Sep 16 18:42:07.299552 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.299536    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"
Sep 16 18:42:07.303332 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.303315    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5p5j4\""
Sep 16 18:42:07.304049 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.304031    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:42:07.324719 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.324692    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"]
Sep 16 18:42:07.443936 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.443900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3c351ff9-597c-44a0-b6c7-ebfba0998ce6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-lf6kg\" (UID: \"3c351ff9-597c-44a0-b6c7-ebfba0998ce6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"
Sep 16 18:42:07.545078 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.545046    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3c351ff9-597c-44a0-b6c7-ebfba0998ce6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-lf6kg\" (UID: \"3c351ff9-597c-44a0-b6c7-ebfba0998ce6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"
Sep 16 18:42:07.547368 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.547348    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3c351ff9-597c-44a0-b6c7-ebfba0998ce6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-lf6kg\" (UID: \"3c351ff9-597c-44a0-b6c7-ebfba0998ce6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"
Sep 16 18:42:07.607516 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.607492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"
Sep 16 18:42:07.896191 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:07.896102    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"]
Sep 16 18:42:07.899687 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:07.899659    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c351ff9_597c_44a0_b6c7_ebfba0998ce6.slice/crio-d4bbfb7e7c82bf944ec12808c988d2e5f522ec0848fc8ef85b3d8696593f0e2d WatchSource:0}: Error finding container d4bbfb7e7c82bf944ec12808c988d2e5f522ec0848fc8ef85b3d8696593f0e2d: Status 404 returned error can't find the container with id d4bbfb7e7c82bf944ec12808c988d2e5f522ec0848fc8ef85b3d8696593f0e2d
Sep 16 18:42:08.203792 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:08.203698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg" event={"ID":"3c351ff9-597c-44a0-b6c7-ebfba0998ce6","Type":"ContainerStarted","Data":"d4bbfb7e7c82bf944ec12808c988d2e5f522ec0848fc8ef85b3d8696593f0e2d"}
Sep 16 18:42:09.538989 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:42:09.538939    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jvtwj" podUID="7be3ea3a-8122-408d-b74f-54c15a936f07"
Sep 16 18:42:09.637218 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:42:09.637172    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7jzmn" podUID="dca28d1e-3d17-4eea-a864-5f984527ae0f"
Sep 16 18:42:10.213018 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:10.212907    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg" event={"ID":"3c351ff9-597c-44a0-b6c7-ebfba0998ce6","Type":"ContainerStarted","Data":"7b39feee940034a7d46f411bdd95c497ef298bc1166dfe5aa7bf66e61a99fd26"}
Sep 16 18:42:10.213018 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:10.212963    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:42:10.240115 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:10.240064    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg" podStartSLOduration=1.8160732529999999 podStartE2EDuration="3.240044505s" totalImagesPullingTime="1.423971252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:10.239862937 +0000 UTC m=+158.056628523" watchObservedRunningTime="2026-09-16 18:42:10.240044505 +0000 UTC m=+158.056810090"
Sep 16 18:42:11.216101 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:11.216069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"
Sep 16 18:42:11.220454 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:11.220429    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lf6kg"
Sep 16 18:42:14.451153 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:14.451112    2575 patch_prober.go:28] interesting pod/image-registry-5998b9f4f4-45jkq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:42:14.451504 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:14.451168    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq" podUID="79199211-4552-4acb-945b-56728721546f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:42:14.497153 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:14.497125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:42:14.499359 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:14.499337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7be3ea3a-8122-408d-b74f-54c15a936f07-metrics-tls\") pod \"dns-default-jvtwj\" (UID: \"7be3ea3a-8122-408d-b74f-54c15a936f07\") " pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:42:14.716971 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:14.716899    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-24cbm\""
Sep 16 18:42:14.724270 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:14.724255    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:42:15.006955 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:15.006882    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jvtwj"]
Sep 16 18:42:15.010580 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:15.010548    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be3ea3a_8122_408d_b74f_54c15a936f07.slice/crio-1c09f90490b31ff167ef649b0a360c54ceeefccbb7ae4f8675a2e31fd6d2b8eb WatchSource:0}: Error finding container 1c09f90490b31ff167ef649b0a360c54ceeefccbb7ae4f8675a2e31fd6d2b8eb: Status 404 returned error can't find the container with id 1c09f90490b31ff167ef649b0a360c54ceeefccbb7ae4f8675a2e31fd6d2b8eb
Sep 16 18:42:15.230197 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:15.230160    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jvtwj" event={"ID":"7be3ea3a-8122-408d-b74f-54c15a936f07","Type":"ContainerStarted","Data":"1c09f90490b31ff167ef649b0a360c54ceeefccbb7ae4f8675a2e31fd6d2b8eb"}
Sep 16 18:42:17.238383 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.238350    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jvtwj" event={"ID":"7be3ea3a-8122-408d-b74f-54c15a936f07","Type":"ContainerStarted","Data":"8fe68e1d33f21cb6a5576cad6c552063e6a0c66b632932b6e1fe73c821522e4d"}
Sep 16 18:42:17.238796 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.238402    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jvtwj" event={"ID":"7be3ea3a-8122-408d-b74f-54c15a936f07","Type":"ContainerStarted","Data":"7a779ba9686d22008bb6c116e21f7fe059de82bcb0229ddedd2b7cf979c36e42"}
Sep 16 18:42:17.238796 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.238458    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:42:17.299392 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.299334    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jvtwj" podStartSLOduration=129.927949944 podStartE2EDuration="2m11.299316792s" totalImagesPullingTime="1.371366848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:17.298782745 +0000 UTC m=+165.115548332" watchObservedRunningTime="2026-09-16 18:42:17.299316792 +0000 UTC m=+165.116082400"
Sep 16 18:42:17.443336 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.443305    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"]
Sep 16 18:42:17.446180 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.446164    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.450608 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.450586    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:42:17.451540 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.451524    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:42:17.452448 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.452427    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:17.452553 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.452465    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:17.452624 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.452557    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:17.452683 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.452658    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:42:17.452760 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.452704    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-trjpt\""
Sep 16 18:42:17.477007 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.476980    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2shq2"]
Sep 16 18:42:17.479726 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.479688    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.482680 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.482658    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:17.482831 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.482815    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:17.483032 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.483011    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xrql4\""
Sep 16 18:42:17.483116 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.483084    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:17.520933 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.520886    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.520933 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.520918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-sys\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.521076 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.520947    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/883516cf-651a-40be-8a75-df98f5018d25-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.521076 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.520966    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/883516cf-651a-40be-8a75-df98f5018d25-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.521076 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521019    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-textfile\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.521076 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/883516cf-651a-40be-8a75-df98f5018d25-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.521221 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzzc\" (UniqueName: \"kubernetes.io/projected/883516cf-651a-40be-8a75-df98f5018d25-kube-api-access-hlzzc\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.521221 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521094    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkn8m\" (UniqueName: \"kubernetes.io/projected/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-kube-api-access-jkn8m\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.521221 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521125    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-metrics-client-ca\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.521221 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/883516cf-651a-40be-8a75-df98f5018d25-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.521221 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521210    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-tls\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.521405 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-wtmp\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.521405 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/883516cf-651a-40be-8a75-df98f5018d25-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.521405 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.521405 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.521279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-root\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.582779 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.582755    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"]
Sep 16 18:42:17.621934 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.621914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-tls\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622031 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.621940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-wtmp\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622031 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.621958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/883516cf-651a-40be-8a75-df98f5018d25-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.622031 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.621977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622031 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-root\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622031 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-sys\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622095    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/883516cf-651a-40be-8a75-df98f5018d25-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622088    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-root\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622102    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-wtmp\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/883516cf-651a-40be-8a75-df98f5018d25-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-sys\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-textfile\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/883516cf-651a-40be-8a75-df98f5018d25-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzzc\" (UniqueName: \"kubernetes.io/projected/883516cf-651a-40be-8a75-df98f5018d25-kube-api-access-hlzzc\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.622280 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkn8m\" (UniqueName: \"kubernetes.io/projected/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-kube-api-access-jkn8m\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622789 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-metrics-client-ca\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622789 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/883516cf-651a-40be-8a75-df98f5018d25-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.622789 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622497    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-textfile\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622789 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622693    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.622789 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622706    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/883516cf-651a-40be-8a75-df98f5018d25-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.622789 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.622765    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/883516cf-651a-40be-8a75-df98f5018d25-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.623187 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.623143    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/883516cf-651a-40be-8a75-df98f5018d25-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.623274 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.623251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-metrics-client-ca\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.624454 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.624437    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.624625 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.624606    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-node-exporter-tls\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.624736 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.624702    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/883516cf-651a-40be-8a75-df98f5018d25-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.624929 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.624912    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/883516cf-651a-40be-8a75-df98f5018d25-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.652116 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.652089    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkn8m\" (UniqueName: \"kubernetes.io/projected/fceb01e6-e703-47fd-ae5b-10fdcd3597a5-kube-api-access-jkn8m\") pod \"node-exporter-2shq2\" (UID: \"fceb01e6-e703-47fd-ae5b-10fdcd3597a5\") " pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.653647 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.653628    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzzc\" (UniqueName: \"kubernetes.io/projected/883516cf-651a-40be-8a75-df98f5018d25-kube-api-access-hlzzc\") pod \"kube-state-metrics-59fbdd77d-8cz97\" (UID: \"883516cf-651a-40be-8a75-df98f5018d25\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.755399 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.755376    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"
Sep 16 18:42:17.787800 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:17.787735    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2shq2"
Sep 16 18:42:17.799331 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:17.799264    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfceb01e6_e703_47fd_ae5b_10fdcd3597a5.slice/crio-584bf1c72a9d940d9d050a3a4697300b33d581612878089486d00c993ac12d1f WatchSource:0}: Error finding container 584bf1c72a9d940d9d050a3a4697300b33d581612878089486d00c993ac12d1f: Status 404 returned error can't find the container with id 584bf1c72a9d940d9d050a3a4697300b33d581612878089486d00c993ac12d1f
Sep 16 18:42:18.095473 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.095444    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97"]
Sep 16 18:42:18.098283 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:18.098255    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883516cf_651a_40be_8a75_df98f5018d25.slice/crio-0d13b4f4c6fb1c5d0391d8db45f57bf4c6c0bd302a493e034e1f842d7ae6fe1a WatchSource:0}: Error finding container 0d13b4f4c6fb1c5d0391d8db45f57bf4c6c0bd302a493e034e1f842d7ae6fe1a: Status 404 returned error can't find the container with id 0d13b4f4c6fb1c5d0391d8db45f57bf4c6c0bd302a493e034e1f842d7ae6fe1a
Sep 16 18:42:18.244770 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.244739    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2shq2" event={"ID":"fceb01e6-e703-47fd-ae5b-10fdcd3597a5","Type":"ContainerStarted","Data":"584bf1c72a9d940d9d050a3a4697300b33d581612878089486d00c993ac12d1f"}
Sep 16 18:42:18.245798 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.245773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97" event={"ID":"883516cf-651a-40be-8a75-df98f5018d25","Type":"ContainerStarted","Data":"0d13b4f4c6fb1c5d0391d8db45f57bf4c6c0bd302a493e034e1f842d7ae6fe1a"}
Sep 16 18:42:18.463530 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.463454    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:18.471929 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.471904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.477542 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.477512    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:42:18.477696 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.477681    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:42:18.477790 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.477761    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:42:18.477790 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.477783    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:42:18.482651 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.482633    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:42:18.486196 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.486179    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:42:18.486315 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.486267    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2gmbg\""
Sep 16 18:42:18.486602 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.486586    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:42:18.496058 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.496038    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:42:18.507850 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.507829    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:42:18.509383 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.509361    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:18.527614 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527591    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.527704 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.527704 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527694    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.527824 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.527872 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-volume\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.527908 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-out\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.527958 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkjq6\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-kube-api-access-pkjq6\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.527958 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527942    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.528030 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527970    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-web-config\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.528030 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.527999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.528030 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.528021    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.528119 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.528055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.528119 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.528083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629312 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629449 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629449 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629449 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-volume\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629449 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629398    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-out\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629449 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkjq6\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-kube-api-access-pkjq6\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629736 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629736 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-web-config\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629736 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629736 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629736 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629736 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.629736 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.629645    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.630983 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.630611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.630983 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.630662    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.631671 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.631222    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.634515 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.634470    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.634617 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.634581    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.635139 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.635093    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.635649 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.635623    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-out\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.636193 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.636169    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-web-config\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.636989 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.636930    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-volume\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.637891 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.637437    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.637891 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.637846    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.638609 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.638567    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.667145 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.667117    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkjq6\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-kube-api-access-pkjq6\") pod \"alertmanager-main-0\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:18.782793 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:18.782771    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:19.091803 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:19.091775    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:19.206463 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:19.206417    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fca5d6_52b0_422b_a859_18afd1d5a7e3.slice/crio-298b81142b8ae8974bc55692b9d64fe0b3658d51846da7e0da9e4660462f8eb9 WatchSource:0}: Error finding container 298b81142b8ae8974bc55692b9d64fe0b3658d51846da7e0da9e4660462f8eb9: Status 404 returned error can't find the container with id 298b81142b8ae8974bc55692b9d64fe0b3658d51846da7e0da9e4660462f8eb9
Sep 16 18:42:19.250402 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:19.250373    2575 generic.go:356] "Generic (PLEG): container finished" podID="fceb01e6-e703-47fd-ae5b-10fdcd3597a5" containerID="ab970e964cd6a27b2fbd6a9c9d5055f2dc99301833f2be2f0a3e5bfc80b1f0df" exitCode=0
Sep 16 18:42:19.250759 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:19.250456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2shq2" event={"ID":"fceb01e6-e703-47fd-ae5b-10fdcd3597a5","Type":"ContainerDied","Data":"ab970e964cd6a27b2fbd6a9c9d5055f2dc99301833f2be2f0a3e5bfc80b1f0df"}
Sep 16 18:42:19.251764 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:19.251742    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerStarted","Data":"298b81142b8ae8974bc55692b9d64fe0b3658d51846da7e0da9e4660462f8eb9"}
Sep 16 18:42:20.258935 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:20.258903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerStarted","Data":"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"}
Sep 16 18:42:20.261391 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:20.261361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2shq2" event={"ID":"fceb01e6-e703-47fd-ae5b-10fdcd3597a5","Type":"ContainerStarted","Data":"9837618666da2b85007db993670dbb2306f4d51c1e3f96131a599a776b24dea2"}
Sep 16 18:42:20.261600 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:20.261411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2shq2" event={"ID":"fceb01e6-e703-47fd-ae5b-10fdcd3597a5","Type":"ContainerStarted","Data":"b4706e7516e6240b426da47688c09620d31f3d510ec0072ba2c1174748e4547c"}
Sep 16 18:42:20.263745 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:20.263705    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97" event={"ID":"883516cf-651a-40be-8a75-df98f5018d25","Type":"ContainerStarted","Data":"49f2795d5f290d9b0d25ba1c76c87dc5d429d6d8495b8dc5db613418e5cbe560"}
Sep 16 18:42:20.263840 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:20.263774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97" event={"ID":"883516cf-651a-40be-8a75-df98f5018d25","Type":"ContainerStarted","Data":"1ef7cbd2e519c99bd45c8473bb183ab8f6c95e26b688f3560a6eb0e42c87d852"}
Sep 16 18:42:20.263840 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:20.263808    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97" event={"ID":"883516cf-651a-40be-8a75-df98f5018d25","Type":"ContainerStarted","Data":"3ae4bb90b11e1fe5d6d8e5864cb3902642f10d6ff461e15569fb62bc04734f3a"}
Sep 16 18:42:20.369310 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:20.369258    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8cz97" podStartSLOduration=2.218700651 podStartE2EDuration="3.369241715s" totalImagesPullingTime="1.150541064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:20.369133442 +0000 UTC m=+168.185899039" watchObservedRunningTime="2026-09-16 18:42:20.369241715 +0000 UTC m=+168.186007297"
Sep 16 18:42:20.400230 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:20.400175    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2shq2" podStartSLOduration=2.387495451 podStartE2EDuration="3.400159414s" totalImagesPullingTime="934.986637ms" totalInitContainerRuntime="77.677326ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:20.398445032 +0000 UTC m=+168.215210619" watchObservedRunningTime="2026-09-16 18:42:20.400159414 +0000 UTC m=+168.216924993"
Sep 16 18:42:21.268977 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.268942    2575 generic.go:356] "Generic (PLEG): container finished" podID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerID="0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a" exitCode=0
Sep 16 18:42:21.269334 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.269032    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerDied","Data":"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"}
Sep 16 18:42:21.348913 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.348881    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-775d8ff5df-n24c5"]
Sep 16 18:42:21.352466 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.352450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.355773 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.355749    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:42:21.367995 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.367971    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6o9a16jhb6jhb\""
Sep 16 18:42:21.369307 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.369292    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:42:21.372346 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.372327    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:42:21.374209 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.374192    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:42:21.375272 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.375255    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:42:21.379387 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.379216    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mmf47\""
Sep 16 18:42:21.391834 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.391806    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-775d8ff5df-n24c5"]
Sep 16 18:42:21.454139 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.454102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.454139 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.454139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfspq\" (UniqueName: \"kubernetes.io/projected/99e4d657-08f2-41f5-bef4-d1a463f56e71-kube-api-access-nfspq\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.454330 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.454212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.454330 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.454234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.454330 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.454281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e4d657-08f2-41f5-bef4-d1a463f56e71-metrics-client-ca\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.454330 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.454310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.454452 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.454347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-tls\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.454452 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.454377    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-grpc-tls\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.554991 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.554910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.554991 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.554948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.554991 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.554972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e4d657-08f2-41f5-bef4-d1a463f56e71-metrics-client-ca\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.555248 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.554998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.555248 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.555045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-tls\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.555248 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.555096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-grpc-tls\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.555248 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.555129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.555248 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.555154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfspq\" (UniqueName: \"kubernetes.io/projected/99e4d657-08f2-41f5-bef4-d1a463f56e71-kube-api-access-nfspq\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.555833 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.555798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e4d657-08f2-41f5-bef4-d1a463f56e71-metrics-client-ca\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.557657 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.557625    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.557929 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.557907    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.557929 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.557919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-grpc-tls\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.558088 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.558071    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.558174 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.558151    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-tls\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.558222 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.558199    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99e4d657-08f2-41f5-bef4-d1a463f56e71-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.576489 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.576463    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfspq\" (UniqueName: \"kubernetes.io/projected/99e4d657-08f2-41f5-bef4-d1a463f56e71-kube-api-access-nfspq\") pod \"thanos-querier-775d8ff5df-n24c5\" (UID: \"99e4d657-08f2-41f5-bef4-d1a463f56e71\") " pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.613666 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.613630    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"]
Sep 16 18:42:21.618193 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.618171    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:21.622166 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.622144    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:42:21.622332 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.622316    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4zn2q\""
Sep 16 18:42:21.627952 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.627928    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"]
Sep 16 18:42:21.656537 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.656504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1332e3a-cd3b-432b-a49b-842d79e355b2-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-9pdnp\" (UID: \"a1332e3a-cd3b-432b-a49b-842d79e355b2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:21.656672 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.656582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8crdl\" (UniqueName: \"kubernetes.io/projected/a1332e3a-cd3b-432b-a49b-842d79e355b2-kube-api-access-8crdl\") pod \"monitoring-plugin-5d85778b9d-9pdnp\" (UID: \"a1332e3a-cd3b-432b-a49b-842d79e355b2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:21.661518 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.661490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:21.757391 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.757360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1332e3a-cd3b-432b-a49b-842d79e355b2-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-9pdnp\" (UID: \"a1332e3a-cd3b-432b-a49b-842d79e355b2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:21.757539 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.757432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8crdl\" (UniqueName: \"kubernetes.io/projected/a1332e3a-cd3b-432b-a49b-842d79e355b2-kube-api-access-8crdl\") pod \"monitoring-plugin-5d85778b9d-9pdnp\" (UID: \"a1332e3a-cd3b-432b-a49b-842d79e355b2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:21.759800 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.759780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a1332e3a-cd3b-432b-a49b-842d79e355b2-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-9pdnp\" (UID: \"a1332e3a-cd3b-432b-a49b-842d79e355b2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:21.771551 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.771522    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8crdl\" (UniqueName: \"kubernetes.io/projected/a1332e3a-cd3b-432b-a49b-842d79e355b2-kube-api-access-8crdl\") pod \"monitoring-plugin-5d85778b9d-9pdnp\" (UID: \"a1332e3a-cd3b-432b-a49b-842d79e355b2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:21.927583 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.927542    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:21.968363 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:21.968332    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-775d8ff5df-n24c5"]
Sep 16 18:42:21.972634 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:21.972591    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e4d657_08f2_41f5_bef4_d1a463f56e71.slice/crio-742aef8a44f081f78ffba1c75d453a363d355915e40ded79bbdcd324de9d3051 WatchSource:0}: Error finding container 742aef8a44f081f78ffba1c75d453a363d355915e40ded79bbdcd324de9d3051: Status 404 returned error can't find the container with id 742aef8a44f081f78ffba1c75d453a363d355915e40ded79bbdcd324de9d3051
Sep 16 18:42:22.025531 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.025504    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-66f699457d-v8h62"]
Sep 16 18:42:22.029947 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.029928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.036652 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.036627    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fniqq4dbb3gtm\""
Sep 16 18:42:22.036783 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.036763    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:42:22.036926 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.036901    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:42:22.037057 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.037040    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:42:22.037212 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.037172    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-l5bgz\""
Sep 16 18:42:22.037312 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.037206    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:42:22.055879 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.055854    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66f699457d-v8h62"]
Sep 16 18:42:22.060315 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.060294    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-secret-metrics-server-tls\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.060421 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.060321    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.060421 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.060347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-secret-metrics-server-client-certs\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.060421 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.060370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-audit-log\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.060574 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.060430    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-client-ca-bundle\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.060574 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.060477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-metrics-server-audit-profiles\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.060574 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.060505    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968jh\" (UniqueName: \"kubernetes.io/projected/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-kube-api-access-968jh\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.161394 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.161360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-secret-metrics-server-tls\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.161547 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.161400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.161547 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.161429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-secret-metrics-server-client-certs\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.161547 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.161454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-audit-log\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.161547 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.161530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-client-ca-bundle\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.161756 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.161592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-metrics-server-audit-profiles\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.161756 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.161631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968jh\" (UniqueName: \"kubernetes.io/projected/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-kube-api-access-968jh\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.161931 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.161903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-audit-log\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.162209 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.162184    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.162627 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.162606    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-metrics-server-audit-profiles\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.163947 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.163928    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-secret-metrics-server-tls\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.164052 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.164032    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-client-ca-bundle\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.164106 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.164086    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-secret-metrics-server-client-certs\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.170750 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.170730    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-968jh\" (UniqueName: \"kubernetes.io/projected/da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99-kube-api-access-968jh\") pod \"metrics-server-66f699457d-v8h62\" (UID: \"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99\") " pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.213136 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.213111    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"]
Sep 16 18:42:22.274332 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.274297    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5" event={"ID":"99e4d657-08f2-41f5-bef4-d1a463f56e71","Type":"ContainerStarted","Data":"742aef8a44f081f78ffba1c75d453a363d355915e40ded79bbdcd324de9d3051"}
Sep 16 18:42:22.345496 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.345463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:22.504972 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:22.504903    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1332e3a_cd3b_432b_a49b_842d79e355b2.slice/crio-abded328beb33008890666d8b2c2a9b2085feb842027f51650e1423c6b1fdf11 WatchSource:0}: Error finding container abded328beb33008890666d8b2c2a9b2085feb842027f51650e1423c6b1fdf11: Status 404 returned error can't find the container with id abded328beb33008890666d8b2c2a9b2085feb842027f51650e1423c6b1fdf11
Sep 16 18:42:22.619256 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.619235    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:42:22.808571 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:22.808547    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66f699457d-v8h62"]
Sep 16 18:42:22.810439 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:22.810410    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5c4ab8_cc5e_4a1b_ad79_f7df6d3b8b99.slice/crio-2ca645162ed1f07a5802b2ac1b7fb55f5e8b9274c369b2fb9d736241f2d0e5d0 WatchSource:0}: Error finding container 2ca645162ed1f07a5802b2ac1b7fb55f5e8b9274c369b2fb9d736241f2d0e5d0: Status 404 returned error can't find the container with id 2ca645162ed1f07a5802b2ac1b7fb55f5e8b9274c369b2fb9d736241f2d0e5d0
Sep 16 18:42:23.283077 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:23.283027    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerStarted","Data":"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"}
Sep 16 18:42:23.283534 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:23.283115    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerStarted","Data":"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"}
Sep 16 18:42:23.283534 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:23.283155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerStarted","Data":"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"}
Sep 16 18:42:23.283534 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:23.283197    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerStarted","Data":"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"}
Sep 16 18:42:23.283534 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:23.283241    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerStarted","Data":"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"}
Sep 16 18:42:23.284666 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:23.284637    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66f699457d-v8h62" event={"ID":"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99","Type":"ContainerStarted","Data":"2ca645162ed1f07a5802b2ac1b7fb55f5e8b9274c369b2fb9d736241f2d0e5d0"}
Sep 16 18:42:23.286024 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:23.286000    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp" event={"ID":"a1332e3a-cd3b-432b-a49b-842d79e355b2","Type":"ContainerStarted","Data":"abded328beb33008890666d8b2c2a9b2085feb842027f51650e1423c6b1fdf11"}
Sep 16 18:42:24.451174 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:24.451104    2575 patch_prober.go:28] interesting pod/image-registry-5998b9f4f4-45jkq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:42:24.451174 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:24.451151    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq" podUID="79199211-4552-4acb-945b-56728721546f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:42:25.296094 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.296056    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp" event={"ID":"a1332e3a-cd3b-432b-a49b-842d79e355b2","Type":"ContainerStarted","Data":"2183a05a00b521ec48d4d92a9f08344376ea5806d77d525738bfd4d7f92b4471"}
Sep 16 18:42:25.296265 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.296239    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:25.300103 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.300079    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerStarted","Data":"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"}
Sep 16 18:42:25.301887 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.301867    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp"
Sep 16 18:42:25.303398 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.303377    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5" event={"ID":"99e4d657-08f2-41f5-bef4-d1a463f56e71","Type":"ContainerStarted","Data":"98b73e1c7ce36701a3b5693c0218bd4e013c21bce97cebddd92f20222a035823"}
Sep 16 18:42:25.303513 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.303431    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5" event={"ID":"99e4d657-08f2-41f5-bef4-d1a463f56e71","Type":"ContainerStarted","Data":"77bdfc98d6decda70fa3108d462faff5c32f6e04540161003ce465ef3cbf3a47"}
Sep 16 18:42:25.303513 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.303477    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:25.303605 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.303519    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5" event={"ID":"99e4d657-08f2-41f5-bef4-d1a463f56e71","Type":"ContainerStarted","Data":"a135bde0617f97fee42375c91cd1e44c44a139caa47195238d2737396b7c6dba"}
Sep 16 18:42:25.303605 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.303564    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5" event={"ID":"99e4d657-08f2-41f5-bef4-d1a463f56e71","Type":"ContainerStarted","Data":"47d5eafd4fc6df3e14e286c4711e5e1093e3f3be970ab636e34b5310ea34dab0"}
Sep 16 18:42:25.303680 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.303603    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5" event={"ID":"99e4d657-08f2-41f5-bef4-d1a463f56e71","Type":"ContainerStarted","Data":"b110ff4d231bbb32fad750c5d1056e379559460c95ed0d7d655d869d4a2990b0"}
Sep 16 18:42:25.303680 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.303648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5" event={"ID":"99e4d657-08f2-41f5-bef4-d1a463f56e71","Type":"ContainerStarted","Data":"cc91d3a28cdacdd39eaeb2992b45c01bfbaf77c5bc9cec08f13891ce52afd029"}
Sep 16 18:42:25.304924 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.304905    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66f699457d-v8h62" event={"ID":"da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99","Type":"ContainerStarted","Data":"dea4aafef5155acb4f0c9e733cf8d163791b5572739590efbb9f5213c561268d"}
Sep 16 18:42:25.324413 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.324360    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-9pdnp" podStartSLOduration=2.592891775 podStartE2EDuration="4.32434463s" totalImagesPullingTime="1.731452855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:25.322551537 +0000 UTC m=+173.139317127" watchObservedRunningTime="2026-09-16 18:42:25.32434463 +0000 UTC m=+173.141110215"
Sep 16 18:42:25.359009 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.358942    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5" podStartSLOduration=2.089293518 podStartE2EDuration="4.358918085s" totalImagesPullingTime="2.269624567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:25.358380028 +0000 UTC m=+173.175145615" watchObservedRunningTime="2026-09-16 18:42:25.358918085 +0000 UTC m=+173.175683672"
Sep 16 18:42:25.394585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.394469    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-66f699457d-v8h62" podStartSLOduration=2.063856847 podStartE2EDuration="4.394446289s" totalImagesPullingTime="2.330589442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:25.386882864 +0000 UTC m=+173.203648440" watchObservedRunningTime="2026-09-16 18:42:25.394446289 +0000 UTC m=+173.211211876"
Sep 16 18:42:25.471347 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:25.471280    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.8207912 podStartE2EDuration="7.471255455s" totalImagesPullingTime="3.635404101s" totalInitContainerRuntime="15.060154ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:25.470011506 +0000 UTC m=+173.286777113" watchObservedRunningTime="2026-09-16 18:42:25.471255455 +0000 UTC m=+173.288021067"
Sep 16 18:42:27.248338 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:27.248300    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jvtwj"
Sep 16 18:42:29.464627 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.464582    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq" podUID="79199211-4552-4acb-945b-56728721546f" containerName="registry" containerID="cri-o://99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad" gracePeriod=30
Sep 16 18:42:29.707952 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.707929    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:42:29.832051 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.831968    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfr85\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-kube-api-access-bfr85\") pod \"79199211-4552-4acb-945b-56728721546f\" (UID: \"79199211-4552-4acb-945b-56728721546f\") "
Sep 16 18:42:29.832051 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.832008    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79199211-4552-4acb-945b-56728721546f-ca-trust-extracted\") pod \"79199211-4552-4acb-945b-56728721546f\" (UID: \"79199211-4552-4acb-945b-56728721546f\") "
Sep 16 18:42:29.832051 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.832038    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-installation-pull-secrets\") pod \"79199211-4552-4acb-945b-56728721546f\" (UID: \"79199211-4552-4acb-945b-56728721546f\") "
Sep 16 18:42:29.832318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.832160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") pod \"79199211-4552-4acb-945b-56728721546f\" (UID: \"79199211-4552-4acb-945b-56728721546f\") "
Sep 16 18:42:29.832318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.832192    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-registry-certificates\") pod \"79199211-4552-4acb-945b-56728721546f\" (UID: \"79199211-4552-4acb-945b-56728721546f\") "
Sep 16 18:42:29.832318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.832214    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-bound-sa-token\") pod \"79199211-4552-4acb-945b-56728721546f\" (UID: \"79199211-4552-4acb-945b-56728721546f\") "
Sep 16 18:42:29.832318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.832245    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-trusted-ca\") pod \"79199211-4552-4acb-945b-56728721546f\" (UID: \"79199211-4552-4acb-945b-56728721546f\") "
Sep 16 18:42:29.832318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.832287    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-image-registry-private-configuration\") pod \"79199211-4552-4acb-945b-56728721546f\" (UID: \"79199211-4552-4acb-945b-56728721546f\") "
Sep 16 18:42:29.832599 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.832574    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "79199211-4552-4acb-945b-56728721546f" (UID: "79199211-4552-4acb-945b-56728721546f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:29.833135 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.833102    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "79199211-4552-4acb-945b-56728721546f" (UID: "79199211-4552-4acb-945b-56728721546f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:29.834557 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.834527    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "79199211-4552-4acb-945b-56728721546f" (UID: "79199211-4552-4acb-945b-56728721546f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:29.834647 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.834550    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "79199211-4552-4acb-945b-56728721546f" (UID: "79199211-4552-4acb-945b-56728721546f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:29.834758 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.834735    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-kube-api-access-bfr85" (OuterVolumeSpecName: "kube-api-access-bfr85") pod "79199211-4552-4acb-945b-56728721546f" (UID: "79199211-4552-4acb-945b-56728721546f"). InnerVolumeSpecName "kube-api-access-bfr85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:29.834823 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.834800    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "79199211-4552-4acb-945b-56728721546f" (UID: "79199211-4552-4acb-945b-56728721546f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:29.834988 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.834972    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "79199211-4552-4acb-945b-56728721546f" (UID: "79199211-4552-4acb-945b-56728721546f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:29.840776 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.840749    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79199211-4552-4acb-945b-56728721546f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "79199211-4552-4acb-945b-56728721546f" (UID: "79199211-4552-4acb-945b-56728721546f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:29.933679 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.933649    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfr85\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-kube-api-access-bfr85\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:42:29.933679 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.933673    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79199211-4552-4acb-945b-56728721546f-ca-trust-extracted\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:42:29.933679 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.933684    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-installation-pull-secrets\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:42:29.933896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.933693    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-registry-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:42:29.933896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.933701    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-registry-certificates\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:42:29.933896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.933726    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79199211-4552-4acb-945b-56728721546f-bound-sa-token\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:42:29.933896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.933734    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79199211-4552-4acb-945b-56728721546f-trusted-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:42:29.933896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:29.933742    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79199211-4552-4acb-945b-56728721546f-image-registry-private-configuration\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:42:30.325262 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.325228    2575 generic.go:356] "Generic (PLEG): container finished" podID="79199211-4552-4acb-945b-56728721546f" containerID="99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad" exitCode=0
Sep 16 18:42:30.325444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.325273    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq"
Sep 16 18:42:30.325444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.325273    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq" event={"ID":"79199211-4552-4acb-945b-56728721546f","Type":"ContainerDied","Data":"99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad"}
Sep 16 18:42:30.325444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.325411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5998b9f4f4-45jkq" event={"ID":"79199211-4552-4acb-945b-56728721546f","Type":"ContainerDied","Data":"d70a45ce48608817c1ef9d3dda1603f70ebfd8bc7cec72251224a40bda641cc1"}
Sep 16 18:42:30.325604 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.325447    2575 scope.go:117] "RemoveContainer" containerID="99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad"
Sep 16 18:42:30.334632 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.334613    2575 scope.go:117] "RemoveContainer" containerID="99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad"
Sep 16 18:42:30.334948 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:42:30.334928    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad\": container with ID starting with 99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad not found: ID does not exist" containerID="99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad"
Sep 16 18:42:30.335019 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.334956    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad"} err="failed to get container status \"99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad\": rpc error: code = NotFound desc = could not find container \"99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad\": container with ID starting with 99716d51f77647e501e5f21551b60b8ebd35659c6776191ae01e957b142283ad not found: ID does not exist"
Sep 16 18:42:30.358607 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.358574    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5998b9f4f4-45jkq"]
Sep 16 18:42:30.369382 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.369355    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5998b9f4f4-45jkq"]
Sep 16 18:42:30.619741 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:30.619695    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79199211-4552-4acb-945b-56728721546f" path="/var/lib/kubelet/pods/79199211-4552-4acb-945b-56728721546f/volumes"
Sep 16 18:42:31.315072 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:31.315043    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-775d8ff5df-n24c5"
Sep 16 18:42:39.481430 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.481397    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84ffd9f457-xvl9j"]
Sep 16 18:42:39.481791 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.481747    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79199211-4552-4acb-945b-56728721546f" containerName="registry"
Sep 16 18:42:39.481791 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.481765    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="79199211-4552-4acb-945b-56728721546f" containerName="registry"
Sep 16 18:42:39.481877 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.481824    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="79199211-4552-4acb-945b-56728721546f" containerName="registry"
Sep 16 18:42:39.487078 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.487055    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.492445 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.492427    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:42:39.492987 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.492967    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:42:39.493091 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.493008    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:42:39.493745 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.493726    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:42:39.493982 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.493965    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:42:39.494069 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.494049    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rwxg4\""
Sep 16 18:42:39.494138 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.494021    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:42:39.494138 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.494025    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:42:39.498926 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.498906    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84ffd9f457-xvl9j"]
Sep 16 18:42:39.613724 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.613681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-console-config\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.613866 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.613729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtjxh\" (UniqueName: \"kubernetes.io/projected/b40bbf02-7283-4a0d-82ed-6881f847e428-kube-api-access-gtjxh\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.613866 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.613754    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-serving-cert\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.613866 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.613837    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-oauth-serving-cert\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.613984 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.613879    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-oauth-config\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.613984 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.613948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-service-ca\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.714256 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.714222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-oauth-config\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.714388 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.714272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-service-ca\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.714388 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.714299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-console-config\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.714388 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.714318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtjxh\" (UniqueName: \"kubernetes.io/projected/b40bbf02-7283-4a0d-82ed-6881f847e428-kube-api-access-gtjxh\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.714388 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.714342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-serving-cert\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.714388 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.714372    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-oauth-serving-cert\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.715088 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.715062    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-console-config\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.715184 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.715090    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-service-ca\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.715184 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.715150    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-oauth-serving-cert\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.716703 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.716682    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-oauth-config\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.716812 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.716789    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-serving-cert\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.724413 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.724394    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtjxh\" (UniqueName: \"kubernetes.io/projected/b40bbf02-7283-4a0d-82ed-6881f847e428-kube-api-access-gtjxh\") pod \"console-84ffd9f457-xvl9j\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") " pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:39.796466 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:39.796400    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:40.075531 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:40.075459    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84ffd9f457-xvl9j"]
Sep 16 18:42:40.078316 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:42:40.078284    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40bbf02_7283_4a0d_82ed_6881f847e428.slice/crio-a61d59d194930589cd4b8c91501f0c248976a0941d1fa65bd241bc932d9e3d7a WatchSource:0}: Error finding container a61d59d194930589cd4b8c91501f0c248976a0941d1fa65bd241bc932d9e3d7a: Status 404 returned error can't find the container with id a61d59d194930589cd4b8c91501f0c248976a0941d1fa65bd241bc932d9e3d7a
Sep 16 18:42:40.366281 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:40.366245    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ffd9f457-xvl9j" event={"ID":"b40bbf02-7283-4a0d-82ed-6881f847e428","Type":"ContainerStarted","Data":"a61d59d194930589cd4b8c91501f0c248976a0941d1fa65bd241bc932d9e3d7a"}
Sep 16 18:42:42.346037 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:42.345991    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:42.346513 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:42.346070    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:42:43.380039 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:43.380005    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ffd9f457-xvl9j" event={"ID":"b40bbf02-7283-4a0d-82ed-6881f847e428","Type":"ContainerStarted","Data":"48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683"}
Sep 16 18:42:43.412896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:43.412843    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84ffd9f457-xvl9j" podStartSLOduration=1.429483371 podStartE2EDuration="4.412826876s" totalImagesPullingTime="2.983343505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:43.412055369 +0000 UTC m=+191.228820956" watchObservedRunningTime="2026-09-16 18:42:43.412826876 +0000 UTC m=+191.229592461"
Sep 16 18:42:46.392857 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:46.392822    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ecea926-c56c-4436-9403-3484377dbfed" containerID="57048607dcc685af9801e7f83bdfb41ae32466440f92bcd081ed90be5bc05ede" exitCode=0
Sep 16 18:42:46.393293 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:46.392898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7" event={"ID":"5ecea926-c56c-4436-9403-3484377dbfed","Type":"ContainerDied","Data":"57048607dcc685af9801e7f83bdfb41ae32466440f92bcd081ed90be5bc05ede"}
Sep 16 18:42:46.393293 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:46.393245    2575 scope.go:117] "RemoveContainer" containerID="57048607dcc685af9801e7f83bdfb41ae32466440f92bcd081ed90be5bc05ede"
Sep 16 18:42:46.394615 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:46.394534    2575 generic.go:356] "Generic (PLEG): container finished" podID="fd80257a-afc9-48f9-b887-6053066bafbf" containerID="32a2e21837b0ed8d0d351af1cc49d3db89fceb9724cfcc0ae9f6d3c6c9fe2b03" exitCode=0
Sep 16 18:42:46.394666 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:46.394612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw" event={"ID":"fd80257a-afc9-48f9-b887-6053066bafbf","Type":"ContainerDied","Data":"32a2e21837b0ed8d0d351af1cc49d3db89fceb9724cfcc0ae9f6d3c6c9fe2b03"}
Sep 16 18:42:46.394924 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:46.394910    2575 scope.go:117] "RemoveContainer" containerID="32a2e21837b0ed8d0d351af1cc49d3db89fceb9724cfcc0ae9f6d3c6c9fe2b03"
Sep 16 18:42:47.401483 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:47.401449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-plmd7" event={"ID":"5ecea926-c56c-4436-9403-3484377dbfed","Type":"ContainerStarted","Data":"220bbce1326815dd15e55b5718d41aec73cbfe6a41e0e8adbdd60b8387cd30aa"}
Sep 16 18:42:47.403778 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:47.403752    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zrcbw" event={"ID":"fd80257a-afc9-48f9-b887-6053066bafbf","Type":"ContainerStarted","Data":"0536b15fc5c70fe29c74bfb441682068ad12e288d5cef32da2ac3e7751bbc6ea"}
Sep 16 18:42:49.796614 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:49.796572    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:49.797013 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:49.796649    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:49.801262 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:49.801242    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:50.419468 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:50.419440    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:42:58.609287 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:58.609248    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c4c9fb988-ncv47_0673ac02-7d60-466c-9f91-bc2f1b19eafb/router/0.log"
Sep 16 18:42:58.628088 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:58.628063    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hb7dq_9bddad80-08a6-44db-9fa4-72fb049e7307/serve-healthcheck-canary/0.log"
Sep 16 18:42:58.712388 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:42:58.712354    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84ffd9f457-xvl9j"]
Sep 16 18:43:02.351731 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:02.351682    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:43:02.355826 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:02.355799    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-66f699457d-v8h62"
Sep 16 18:43:23.732955 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:23.732902    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84ffd9f457-xvl9j" podUID="b40bbf02-7283-4a0d-82ed-6881f847e428" containerName="console" containerID="cri-o://48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683" gracePeriod=15
Sep 16 18:43:23.965335 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:23.965313    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:43:24.096631 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.096606    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtjxh\" (UniqueName: \"kubernetes.io/projected/b40bbf02-7283-4a0d-82ed-6881f847e428-kube-api-access-gtjxh\") pod \"b40bbf02-7283-4a0d-82ed-6881f847e428\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") "
Sep 16 18:43:24.096806 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.096645    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-serving-cert\") pod \"b40bbf02-7283-4a0d-82ed-6881f847e428\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") "
Sep 16 18:43:24.096806 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.096676    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-oauth-serving-cert\") pod \"b40bbf02-7283-4a0d-82ed-6881f847e428\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") "
Sep 16 18:43:24.096806 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.096692    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-service-ca\") pod \"b40bbf02-7283-4a0d-82ed-6881f847e428\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") "
Sep 16 18:43:24.096806 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.096743    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-console-config\") pod \"b40bbf02-7283-4a0d-82ed-6881f847e428\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") "
Sep 16 18:43:24.096806 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.096788    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-oauth-config\") pod \"b40bbf02-7283-4a0d-82ed-6881f847e428\" (UID: \"b40bbf02-7283-4a0d-82ed-6881f847e428\") "
Sep 16 18:43:24.097131 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.097098    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b40bbf02-7283-4a0d-82ed-6881f847e428" (UID: "b40bbf02-7283-4a0d-82ed-6881f847e428"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:24.097228 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.097159    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-console-config" (OuterVolumeSpecName: "console-config") pod "b40bbf02-7283-4a0d-82ed-6881f847e428" (UID: "b40bbf02-7283-4a0d-82ed-6881f847e428"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:24.097275 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.097229    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-service-ca" (OuterVolumeSpecName: "service-ca") pod "b40bbf02-7283-4a0d-82ed-6881f847e428" (UID: "b40bbf02-7283-4a0d-82ed-6881f847e428"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:24.100758 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.099048    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b40bbf02-7283-4a0d-82ed-6881f847e428" (UID: "b40bbf02-7283-4a0d-82ed-6881f847e428"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:24.100758 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.099397    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b40bbf02-7283-4a0d-82ed-6881f847e428" (UID: "b40bbf02-7283-4a0d-82ed-6881f847e428"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:24.104610 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.104411    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40bbf02-7283-4a0d-82ed-6881f847e428-kube-api-access-gtjxh" (OuterVolumeSpecName: "kube-api-access-gtjxh") pod "b40bbf02-7283-4a0d-82ed-6881f847e428" (UID: "b40bbf02-7283-4a0d-82ed-6881f847e428"). InnerVolumeSpecName "kube-api-access-gtjxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:24.198415 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.198391    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtjxh\" (UniqueName: \"kubernetes.io/projected/b40bbf02-7283-4a0d-82ed-6881f847e428-kube-api-access-gtjxh\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:43:24.198415 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.198413    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-serving-cert\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:43:24.198528 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.198422    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-oauth-serving-cert\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:43:24.198528 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.198431    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:43:24.198528 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.198440    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b40bbf02-7283-4a0d-82ed-6881f847e428-console-config\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:43:24.198528 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.198448    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b40bbf02-7283-4a0d-82ed-6881f847e428-console-oauth-config\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:43:24.548768 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.548657    2575 generic.go:356] "Generic (PLEG): container finished" podID="b40bbf02-7283-4a0d-82ed-6881f847e428" containerID="48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683" exitCode=0
Sep 16 18:43:24.548768 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.548732    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84ffd9f457-xvl9j"
Sep 16 18:43:24.548768 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.548751    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ffd9f457-xvl9j" event={"ID":"b40bbf02-7283-4a0d-82ed-6881f847e428","Type":"ContainerDied","Data":"48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683"}
Sep 16 18:43:24.549040 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.548844    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84ffd9f457-xvl9j" event={"ID":"b40bbf02-7283-4a0d-82ed-6881f847e428","Type":"ContainerDied","Data":"a61d59d194930589cd4b8c91501f0c248976a0941d1fa65bd241bc932d9e3d7a"}
Sep 16 18:43:24.549040 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.548874    2575 scope.go:117] "RemoveContainer" containerID="48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683"
Sep 16 18:43:24.557877 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.557858    2575 scope.go:117] "RemoveContainer" containerID="48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683"
Sep 16 18:43:24.558141 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:43:24.558122    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683\": container with ID starting with 48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683 not found: ID does not exist" containerID="48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683"
Sep 16 18:43:24.558202 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.558147    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683"} err="failed to get container status \"48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683\": rpc error: code = NotFound desc = could not find container \"48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683\": container with ID starting with 48bb1e8e85904d8a639163b18a04eb005a9b7711a82363ec0530e6af751a3683 not found: ID does not exist"
Sep 16 18:43:24.572973 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.572952    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84ffd9f457-xvl9j"]
Sep 16 18:43:24.578029 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.578011    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84ffd9f457-xvl9j"]
Sep 16 18:43:24.619137 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:24.619115    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b40bbf02-7283-4a0d-82ed-6881f847e428" path="/var/lib/kubelet/pods/b40bbf02-7283-4a0d-82ed-6881f847e428/volumes"
Sep 16 18:43:44.462452 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:44.462418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:43:44.464749 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:44.464729    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca28d1e-3d17-4eea-a864-5f984527ae0f-metrics-certs\") pod \"network-metrics-daemon-7jzmn\" (UID: \"dca28d1e-3d17-4eea-a864-5f984527ae0f\") " pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:43:44.523396 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:44.523368    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l6kxj\""
Sep 16 18:43:44.530698 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:44.530679    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jzmn"
Sep 16 18:43:44.806750 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:44.806662    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7jzmn"]
Sep 16 18:43:44.809648 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:43:44.809622    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca28d1e_3d17_4eea_a864_5f984527ae0f.slice/crio-df92210bc3fa28f6cab94dc7ccb411d3caa17d5b2e195de6831b05f56749566a WatchSource:0}: Error finding container df92210bc3fa28f6cab94dc7ccb411d3caa17d5b2e195de6831b05f56749566a: Status 404 returned error can't find the container with id df92210bc3fa28f6cab94dc7ccb411d3caa17d5b2e195de6831b05f56749566a
Sep 16 18:43:45.629828 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:45.629792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7jzmn" event={"ID":"dca28d1e-3d17-4eea-a864-5f984527ae0f","Type":"ContainerStarted","Data":"df92210bc3fa28f6cab94dc7ccb411d3caa17d5b2e195de6831b05f56749566a"}
Sep 16 18:43:46.635658 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:46.635623    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7jzmn" event={"ID":"dca28d1e-3d17-4eea-a864-5f984527ae0f","Type":"ContainerStarted","Data":"7ab4658f3e191f2d74fa1cccc2e02a9ccb7733a0bf840d79d369eb3e92dee6f7"}
Sep 16 18:43:46.636061 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:46.635678    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7jzmn" event={"ID":"dca28d1e-3d17-4eea-a864-5f984527ae0f","Type":"ContainerStarted","Data":"1894d860e4d4796c4805245bcd0d8e1153e6535b4c095ecab0ca1cd587c01930"}
Sep 16 18:43:46.660590 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:43:46.660537    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7jzmn" podStartSLOduration=252.702332906 podStartE2EDuration="4m13.660520367s" totalImagesPullingTime="958.187461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:46.660148287 +0000 UTC m=+254.476913872" watchObservedRunningTime="2026-09-16 18:43:46.660520367 +0000 UTC m=+254.477285954"
Sep 16 18:44:32.544886 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:44:32.544856    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:47:23.215565 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.215480    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:23.216092 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.215768    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="alertmanager" containerID="cri-o://6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a" gracePeriod=120
Sep 16 18:47:23.216092 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.215797    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy-web" containerID="cri-o://21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28" gracePeriod=120
Sep 16 18:47:23.216092 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.215794    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="config-reloader" containerID="cri-o://b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36" gracePeriod=120
Sep 16 18:47:23.216092 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.215847    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy" containerID="cri-o://b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf" gracePeriod=120
Sep 16 18:47:23.216092 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.215780    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy-metric" containerID="cri-o://60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58" gracePeriod=120
Sep 16 18:47:23.216092 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.215806    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="prom-label-proxy" containerID="cri-o://676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144" gracePeriod=120
Sep 16 18:47:23.456946 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.456920    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:23.458018 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.457997    2575 generic.go:356] "Generic (PLEG): container finished" podID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerID="676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144" exitCode=0
Sep 16 18:47:23.458018 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458017    2575 generic.go:356] "Generic (PLEG): container finished" podID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerID="60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58" exitCode=0
Sep 16 18:47:23.458137 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458024    2575 generic.go:356] "Generic (PLEG): container finished" podID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerID="b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf" exitCode=0
Sep 16 18:47:23.458137 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458029    2575 generic.go:356] "Generic (PLEG): container finished" podID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerID="21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28" exitCode=0
Sep 16 18:47:23.458137 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458035    2575 generic.go:356] "Generic (PLEG): container finished" podID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerID="b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36" exitCode=0
Sep 16 18:47:23.458137 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458040    2575 generic.go:356] "Generic (PLEG): container finished" podID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerID="6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a" exitCode=0
Sep 16 18:47:23.458137 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458065    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerDied","Data":"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"}
Sep 16 18:47:23.458283 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458159    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerDied","Data":"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"}
Sep 16 18:47:23.458283 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458193    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerDied","Data":"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"}
Sep 16 18:47:23.458283 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerDied","Data":"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"}
Sep 16 18:47:23.458283 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458225    2575 scope.go:117] "RemoveContainer" containerID="676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"
Sep 16 18:47:23.458283 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458246    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerDied","Data":"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"}
Sep 16 18:47:23.458283 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerDied","Data":"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"}
Sep 16 18:47:23.458476 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.458303    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"79fca5d6-52b0-422b-a859-18afd1d5a7e3","Type":"ContainerDied","Data":"298b81142b8ae8974bc55692b9d64fe0b3658d51846da7e0da9e4660462f8eb9"}
Sep 16 18:47:23.465845 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.465758    2575 scope.go:117] "RemoveContainer" containerID="60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"
Sep 16 18:47:23.472356 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.472341    2575 scope.go:117] "RemoveContainer" containerID="b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"
Sep 16 18:47:23.478776 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.478759    2575 scope.go:117] "RemoveContainer" containerID="21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"
Sep 16 18:47:23.485175 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.485155    2575 scope.go:117] "RemoveContainer" containerID="b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"
Sep 16 18:47:23.491824 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.491781    2575 scope.go:117] "RemoveContainer" containerID="6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"
Sep 16 18:47:23.499973 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.499949    2575 scope.go:117] "RemoveContainer" containerID="0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"
Sep 16 18:47:23.508121 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.508103    2575 scope.go:117] "RemoveContainer" containerID="676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"
Sep 16 18:47:23.508559 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:47:23.508509    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": container with ID starting with 676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144 not found: ID does not exist" containerID="676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"
Sep 16 18:47:23.508661 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.508550    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"} err="failed to get container status \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": rpc error: code = NotFound desc = could not find container \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": container with ID starting with 676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144 not found: ID does not exist"
Sep 16 18:47:23.508661 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.508579    2575 scope.go:117] "RemoveContainer" containerID="60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"
Sep 16 18:47:23.508956 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:47:23.508932    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": container with ID starting with 60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58 not found: ID does not exist" containerID="60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"
Sep 16 18:47:23.509008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.508966    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"} err="failed to get container status \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": rpc error: code = NotFound desc = could not find container \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": container with ID starting with 60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58 not found: ID does not exist"
Sep 16 18:47:23.509008 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.508989    2575 scope.go:117] "RemoveContainer" containerID="b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"
Sep 16 18:47:23.509300 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:47:23.509281    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": container with ID starting with b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf not found: ID does not exist" containerID="b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"
Sep 16 18:47:23.509362 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.509308    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"} err="failed to get container status \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": rpc error: code = NotFound desc = could not find container \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": container with ID starting with b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf not found: ID does not exist"
Sep 16 18:47:23.509362 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.509325    2575 scope.go:117] "RemoveContainer" containerID="21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"
Sep 16 18:47:23.509596 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:47:23.509579    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": container with ID starting with 21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28 not found: ID does not exist" containerID="21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"
Sep 16 18:47:23.509658 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.509600    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"} err="failed to get container status \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": rpc error: code = NotFound desc = could not find container \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": container with ID starting with 21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28 not found: ID does not exist"
Sep 16 18:47:23.509658 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.509613    2575 scope.go:117] "RemoveContainer" containerID="b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"
Sep 16 18:47:23.509932 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:47:23.509913    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": container with ID starting with b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36 not found: ID does not exist" containerID="b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"
Sep 16 18:47:23.510013 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.509934    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"} err="failed to get container status \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": rpc error: code = NotFound desc = could not find container \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": container with ID starting with b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36 not found: ID does not exist"
Sep 16 18:47:23.510013 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.509952    2575 scope.go:117] "RemoveContainer" containerID="6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"
Sep 16 18:47:23.510241 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:47:23.510215    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": container with ID starting with 6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a not found: ID does not exist" containerID="6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"
Sep 16 18:47:23.510282 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.510253    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"} err="failed to get container status \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": rpc error: code = NotFound desc = could not find container \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": container with ID starting with 6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a not found: ID does not exist"
Sep 16 18:47:23.510318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.510278    2575 scope.go:117] "RemoveContainer" containerID="0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"
Sep 16 18:47:23.510565 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:47:23.510548    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": container with ID starting with 0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a not found: ID does not exist" containerID="0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"
Sep 16 18:47:23.510635 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.510570    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"} err="failed to get container status \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": rpc error: code = NotFound desc = could not find container \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": container with ID starting with 0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a not found: ID does not exist"
Sep 16 18:47:23.510635 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.510585    2575 scope.go:117] "RemoveContainer" containerID="676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"
Sep 16 18:47:23.510981 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.510946    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"} err="failed to get container status \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": rpc error: code = NotFound desc = could not find container \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": container with ID starting with 676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144 not found: ID does not exist"
Sep 16 18:47:23.510981 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.510969    2575 scope.go:117] "RemoveContainer" containerID="60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"
Sep 16 18:47:23.511228 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.511211    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"} err="failed to get container status \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": rpc error: code = NotFound desc = could not find container \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": container with ID starting with 60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58 not found: ID does not exist"
Sep 16 18:47:23.511294 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.511228    2575 scope.go:117] "RemoveContainer" containerID="b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"
Sep 16 18:47:23.511523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.511498    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"} err="failed to get container status \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": rpc error: code = NotFound desc = could not find container \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": container with ID starting with b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf not found: ID does not exist"
Sep 16 18:47:23.511523 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.511521    2575 scope.go:117] "RemoveContainer" containerID="21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"
Sep 16 18:47:23.511828 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.511811    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"} err="failed to get container status \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": rpc error: code = NotFound desc = could not find container \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": container with ID starting with 21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28 not found: ID does not exist"
Sep 16 18:47:23.511828 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.511829    2575 scope.go:117] "RemoveContainer" containerID="b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"
Sep 16 18:47:23.512110 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.512083    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"} err="failed to get container status \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": rpc error: code = NotFound desc = could not find container \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": container with ID starting with b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36 not found: ID does not exist"
Sep 16 18:47:23.512191 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.512112    2575 scope.go:117] "RemoveContainer" containerID="6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"
Sep 16 18:47:23.512396 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.512380    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"} err="failed to get container status \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": rpc error: code = NotFound desc = could not find container \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": container with ID starting with 6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a not found: ID does not exist"
Sep 16 18:47:23.512396 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.512396    2575 scope.go:117] "RemoveContainer" containerID="0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"
Sep 16 18:47:23.512670 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.512651    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"} err="failed to get container status \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": rpc error: code = NotFound desc = could not find container \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": container with ID starting with 0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a not found: ID does not exist"
Sep 16 18:47:23.512760 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.512672    2575 scope.go:117] "RemoveContainer" containerID="676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"
Sep 16 18:47:23.512955 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.512936    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"} err="failed to get container status \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": rpc error: code = NotFound desc = could not find container \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": container with ID starting with 676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144 not found: ID does not exist"
Sep 16 18:47:23.513056 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.512956    2575 scope.go:117] "RemoveContainer" containerID="60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"
Sep 16 18:47:23.513233 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.513215    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"} err="failed to get container status \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": rpc error: code = NotFound desc = could not find container \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": container with ID starting with 60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58 not found: ID does not exist"
Sep 16 18:47:23.513233 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.513232    2575 scope.go:117] "RemoveContainer" containerID="b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"
Sep 16 18:47:23.513453 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.513438    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"} err="failed to get container status \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": rpc error: code = NotFound desc = could not find container \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": container with ID starting with b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf not found: ID does not exist"
Sep 16 18:47:23.513453 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.513452    2575 scope.go:117] "RemoveContainer" containerID="21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"
Sep 16 18:47:23.513705 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.513681    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"} err="failed to get container status \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": rpc error: code = NotFound desc = could not find container \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": container with ID starting with 21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28 not found: ID does not exist"
Sep 16 18:47:23.513796 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.513754    2575 scope.go:117] "RemoveContainer" containerID="b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"
Sep 16 18:47:23.514015 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.513998    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"} err="failed to get container status \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": rpc error: code = NotFound desc = could not find container \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": container with ID starting with b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36 not found: ID does not exist"
Sep 16 18:47:23.514015 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.514014    2575 scope.go:117] "RemoveContainer" containerID="6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"
Sep 16 18:47:23.514226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.514211    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"} err="failed to get container status \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": rpc error: code = NotFound desc = could not find container \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": container with ID starting with 6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a not found: ID does not exist"
Sep 16 18:47:23.514226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.514225    2575 scope.go:117] "RemoveContainer" containerID="0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"
Sep 16 18:47:23.514451 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.514430    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"} err="failed to get container status \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": rpc error: code = NotFound desc = could not find container \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": container with ID starting with 0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a not found: ID does not exist"
Sep 16 18:47:23.514514 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.514452    2575 scope.go:117] "RemoveContainer" containerID="676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"
Sep 16 18:47:23.514690 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.514673    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"} err="failed to get container status \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": rpc error: code = NotFound desc = could not find container \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": container with ID starting with 676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144 not found: ID does not exist"
Sep 16 18:47:23.514771 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.514692    2575 scope.go:117] "RemoveContainer" containerID="60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"
Sep 16 18:47:23.515006 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.514989    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"} err="failed to get container status \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": rpc error: code = NotFound desc = could not find container \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": container with ID starting with 60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58 not found: ID does not exist"
Sep 16 18:47:23.515073 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.515007    2575 scope.go:117] "RemoveContainer" containerID="b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"
Sep 16 18:47:23.515236 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.515221    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"} err="failed to get container status \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": rpc error: code = NotFound desc = could not find container \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": container with ID starting with b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf not found: ID does not exist"
Sep 16 18:47:23.515296 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.515235    2575 scope.go:117] "RemoveContainer" containerID="21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"
Sep 16 18:47:23.515453 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.515435    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"} err="failed to get container status \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": rpc error: code = NotFound desc = could not find container \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": container with ID starting with 21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28 not found: ID does not exist"
Sep 16 18:47:23.515515 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.515453    2575 scope.go:117] "RemoveContainer" containerID="b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"
Sep 16 18:47:23.515681 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.515665    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"} err="failed to get container status \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": rpc error: code = NotFound desc = could not find container \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": container with ID starting with b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36 not found: ID does not exist"
Sep 16 18:47:23.515778 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.515680    2575 scope.go:117] "RemoveContainer" containerID="6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"
Sep 16 18:47:23.515928 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.515909    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"} err="failed to get container status \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": rpc error: code = NotFound desc = could not find container \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": container with ID starting with 6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a not found: ID does not exist"
Sep 16 18:47:23.515987 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.515929    2575 scope.go:117] "RemoveContainer" containerID="0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"
Sep 16 18:47:23.516161 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.516143    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"} err="failed to get container status \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": rpc error: code = NotFound desc = could not find container \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": container with ID starting with 0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a not found: ID does not exist"
Sep 16 18:47:23.516226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.516160    2575 scope.go:117] "RemoveContainer" containerID="676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"
Sep 16 18:47:23.516398 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.516380    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"} err="failed to get container status \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": rpc error: code = NotFound desc = could not find container \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": container with ID starting with 676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144 not found: ID does not exist"
Sep 16 18:47:23.516461 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.516399    2575 scope.go:117] "RemoveContainer" containerID="60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"
Sep 16 18:47:23.516650 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.516621    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"} err="failed to get container status \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": rpc error: code = NotFound desc = could not find container \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": container with ID starting with 60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58 not found: ID does not exist"
Sep 16 18:47:23.516741 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.516650    2575 scope.go:117] "RemoveContainer" containerID="b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"
Sep 16 18:47:23.516929 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.516908    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"} err="failed to get container status \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": rpc error: code = NotFound desc = could not find container \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": container with ID starting with b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf not found: ID does not exist"
Sep 16 18:47:23.516929 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.516930    2575 scope.go:117] "RemoveContainer" containerID="21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"
Sep 16 18:47:23.517201 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.517173    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"} err="failed to get container status \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": rpc error: code = NotFound desc = could not find container \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": container with ID starting with 21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28 not found: ID does not exist"
Sep 16 18:47:23.517257 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.517202    2575 scope.go:117] "RemoveContainer" containerID="b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"
Sep 16 18:47:23.517428 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.517413    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"} err="failed to get container status \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": rpc error: code = NotFound desc = could not find container \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": container with ID starting with b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36 not found: ID does not exist"
Sep 16 18:47:23.517478 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.517429    2575 scope.go:117] "RemoveContainer" containerID="6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"
Sep 16 18:47:23.517635 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.517619    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"} err="failed to get container status \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": rpc error: code = NotFound desc = could not find container \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": container with ID starting with 6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a not found: ID does not exist"
Sep 16 18:47:23.517676 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.517635    2575 scope.go:117] "RemoveContainer" containerID="0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"
Sep 16 18:47:23.517890 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.517869    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"} err="failed to get container status \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": rpc error: code = NotFound desc = could not find container \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": container with ID starting with 0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a not found: ID does not exist"
Sep 16 18:47:23.517890 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.517890    2575 scope.go:117] "RemoveContainer" containerID="676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"
Sep 16 18:47:23.518163 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.518138    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144"} err="failed to get container status \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": rpc error: code = NotFound desc = could not find container \"676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144\": container with ID starting with 676b8cc51b444820c4c088e9a14566bbda2288b4f936493eaab0eed8238a9144 not found: ID does not exist"
Sep 16 18:47:23.518163 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.518163    2575 scope.go:117] "RemoveContainer" containerID="60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"
Sep 16 18:47:23.518414 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.518385    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58"} err="failed to get container status \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": rpc error: code = NotFound desc = could not find container \"60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58\": container with ID starting with 60874797118f91ec5215058999a17032d3fefcc0bd50616b711b2edcdf861f58 not found: ID does not exist"
Sep 16 18:47:23.518414 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.518406    2575 scope.go:117] "RemoveContainer" containerID="b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"
Sep 16 18:47:23.518665 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.518645    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf"} err="failed to get container status \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": rpc error: code = NotFound desc = could not find container \"b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf\": container with ID starting with b094ef353cf4d0ad4f11ae90c46342614b2866afaaa430e97c0e4c5ff845beaf not found: ID does not exist"
Sep 16 18:47:23.518744 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.518665    2575 scope.go:117] "RemoveContainer" containerID="21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"
Sep 16 18:47:23.518899 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.518883    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28"} err="failed to get container status \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": rpc error: code = NotFound desc = could not find container \"21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28\": container with ID starting with 21149f1f8736faa3897ff43c4bbe90fef35a27a5704518eda375f28587d83a28 not found: ID does not exist"
Sep 16 18:47:23.518965 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.518900    2575 scope.go:117] "RemoveContainer" containerID="b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"
Sep 16 18:47:23.519097 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.519080    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36"} err="failed to get container status \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": rpc error: code = NotFound desc = could not find container \"b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36\": container with ID starting with b95a7f769b3e904658bd27f5ca3de99c9e4a4a24016872be293b094337f8da36 not found: ID does not exist"
Sep 16 18:47:23.519158 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.519097    2575 scope.go:117] "RemoveContainer" containerID="6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"
Sep 16 18:47:23.519301 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.519283    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a"} err="failed to get container status \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": rpc error: code = NotFound desc = could not find container \"6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a\": container with ID starting with 6f46d48ec44943b21916542dee54c8f7460152f7dcd8da345b341494aca02c9a not found: ID does not exist"
Sep 16 18:47:23.519355 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.519303    2575 scope.go:117] "RemoveContainer" containerID="0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"
Sep 16 18:47:23.519535 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.519517    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a"} err="failed to get container status \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": rpc error: code = NotFound desc = could not find container \"0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a\": container with ID starting with 0c595efb71b6c7e0e40da914e4b5b0af7cd6a0a05b7f2cc0cca0726d5058119a not found: ID does not exist"
Sep 16 18:47:23.546407 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546376    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkjq6\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-kube-api-access-pkjq6\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546510 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546423    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-out\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546510 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546450    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-volume\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546510 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546484    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546663 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546536    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-metrics-client-ca\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546663 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546564    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-web-config\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546663 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546591    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-main-db\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546663 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546631    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-main-tls\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546888 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546685    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-trusted-ca-bundle\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546888 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546722    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-cluster-tls-config\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546888 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546751    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546888 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546777    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-tls-assets\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.546888 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.546812    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\" (UID: \"79fca5d6-52b0-422b-a859-18afd1d5a7e3\") "
Sep 16 18:47:23.547268 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.547239    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:23.548314 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.547664    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:23.548314 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.548042    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:23.550696 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.550646    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-volume" (OuterVolumeSpecName: "config-volume") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:23.550991 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.550770    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:23.550991 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.550835    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-out" (OuterVolumeSpecName: "config-out") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:23.550991 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.550844    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-kube-api-access-pkjq6" (OuterVolumeSpecName: "kube-api-access-pkjq6") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "kube-api-access-pkjq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:23.551243 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.551218    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:23.551344 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.551252    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:23.551344 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.551287    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:23.551344 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.551316    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:23.555057 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.555031    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:23.560574 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.560554    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-web-config" (OuterVolumeSpecName: "web-config") pod "79fca5d6-52b0-422b-a859-18afd1d5a7e3" (UID: "79fca5d6-52b0-422b-a859-18afd1d5a7e3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:23.648015 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.647975    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648015 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648012    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-metrics-client-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648039    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-web-config\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648048    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-main-db\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648057    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-main-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648065    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79fca5d6-52b0-422b-a859-18afd1d5a7e3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648074    2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-cluster-tls-config\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648082    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648090    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-tls-assets\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648098    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648107    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkjq6\" (UniqueName: \"kubernetes.io/projected/79fca5d6-52b0-422b-a859-18afd1d5a7e3-kube-api-access-pkjq6\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648115    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-out\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:23.648171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:23.648123    2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/79fca5d6-52b0-422b-a859-18afd1d5a7e3-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:47:24.462748 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.462720    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.486977 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.486921    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:24.490101 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.490075    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:24.531226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531197    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:24.531480 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531466    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy-web"
Sep 16 18:47:24.531529 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531482    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy-web"
Sep 16 18:47:24.531529 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531494    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy-metric"
Sep 16 18:47:24.531529 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531500    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy-metric"
Sep 16 18:47:24.531529 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531511    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b40bbf02-7283-4a0d-82ed-6881f847e428" containerName="console"
Sep 16 18:47:24.531529 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531517    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b40bbf02-7283-4a0d-82ed-6881f847e428" containerName="console"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531531    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531540    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531549    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="alertmanager"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531555    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="alertmanager"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531563    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="init-config-reloader"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531569    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="init-config-reloader"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531574    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="prom-label-proxy"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531579    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="prom-label-proxy"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531586    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="config-reloader"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531591    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="config-reloader"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531636    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="config-reloader"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531644    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy-web"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531650    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="alertmanager"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531654    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="prom-label-proxy"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531660    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531665    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" containerName="kube-rbac-proxy-metric"
Sep 16 18:47:24.531673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.531669    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b40bbf02-7283-4a0d-82ed-6881f847e428" containerName="console"
Sep 16 18:47:24.536685 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.536666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.539832 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.539813    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:47:24.541276 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.541254    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:47:24.541366 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.541335    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2gmbg\""
Sep 16 18:47:24.541848 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.541830    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:47:24.542050 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.542023    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:47:24.542115 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.542053    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:47:24.542115 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.542106    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:47:24.542219 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.542114    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:47:24.542219 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.542106    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:47:24.548175 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.548158    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:47:24.553982 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.553963    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:24.619416 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.619392    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79fca5d6-52b0-422b-a859-18afd1d5a7e3" path="/var/lib/kubelet/pods/79fca5d6-52b0-422b-a859-18afd1d5a7e3/volumes"
Sep 16 18:47:24.656810 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.656785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2eb99315-c9d5-47fc-a7d0-2998af266706-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.656917 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.656818    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2eb99315-c9d5-47fc-a7d0-2998af266706-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.656917 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.656849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-config-volume\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657026 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.656915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657026 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.656960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xgl7\" (UniqueName: \"kubernetes.io/projected/2eb99315-c9d5-47fc-a7d0-2998af266706-kube-api-access-5xgl7\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657026 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.656991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eb99315-c9d5-47fc-a7d0-2998af266706-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657180 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.657048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657180 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.657079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2eb99315-c9d5-47fc-a7d0-2998af266706-config-out\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657180 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.657103    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657180 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.657137    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-web-config\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657180 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.657169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eb99315-c9d5-47fc-a7d0-2998af266706-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657369 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.657198    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.657369 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.657227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758349 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758349 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758349 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2eb99315-c9d5-47fc-a7d0-2998af266706-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758537 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2eb99315-c9d5-47fc-a7d0-2998af266706-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758537 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-config-volume\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758537 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758537 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgl7\" (UniqueName: \"kubernetes.io/projected/2eb99315-c9d5-47fc-a7d0-2998af266706-kube-api-access-5xgl7\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758537 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eb99315-c9d5-47fc-a7d0-2998af266706-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758808 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758808 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758604    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2eb99315-c9d5-47fc-a7d0-2998af266706-config-out\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758808 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758808 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-web-config\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758808 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758728    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eb99315-c9d5-47fc-a7d0-2998af266706-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.758808 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.758739    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2eb99315-c9d5-47fc-a7d0-2998af266706-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.759437 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.759412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eb99315-c9d5-47fc-a7d0-2998af266706-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.761415 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.761390    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2eb99315-c9d5-47fc-a7d0-2998af266706-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.761545 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.761521    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.761620 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.761559    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-config-volume\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.761695 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.761679    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.761802 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.761783    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.762021 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.762000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-web-config\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.762266 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.762244    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eb99315-c9d5-47fc-a7d0-2998af266706-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.762342 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.762302    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.762536 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.762515    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2eb99315-c9d5-47fc-a7d0-2998af266706-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.763358 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.763340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2eb99315-c9d5-47fc-a7d0-2998af266706-config-out\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.767127 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.767106    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgl7\" (UniqueName: \"kubernetes.io/projected/2eb99315-c9d5-47fc-a7d0-2998af266706-kube-api-access-5xgl7\") pod \"alertmanager-main-0\" (UID: \"2eb99315-c9d5-47fc-a7d0-2998af266706\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:24.845539 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:24.845514    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:25.128124 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:25.128099    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:25.130637 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:47:25.130610    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb99315_c9d5_47fc_a7d0_2998af266706.slice/crio-652ce7ff58bc4676118163af56f46e3da8d285982148ce0a3a7f8a9f39d9d12d WatchSource:0}: Error finding container 652ce7ff58bc4676118163af56f46e3da8d285982148ce0a3a7f8a9f39d9d12d: Status 404 returned error can't find the container with id 652ce7ff58bc4676118163af56f46e3da8d285982148ce0a3a7f8a9f39d9d12d
Sep 16 18:47:25.474050 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:25.473957    2575 generic.go:356] "Generic (PLEG): container finished" podID="2eb99315-c9d5-47fc-a7d0-2998af266706" containerID="6f2b485cc61d5c2c0f9b37f4858fdcb2fab900fe572f6270c106a9c35ae8c31b" exitCode=0
Sep 16 18:47:25.474050 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:25.474008    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2eb99315-c9d5-47fc-a7d0-2998af266706","Type":"ContainerDied","Data":"6f2b485cc61d5c2c0f9b37f4858fdcb2fab900fe572f6270c106a9c35ae8c31b"}
Sep 16 18:47:25.474436 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:25.474082    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2eb99315-c9d5-47fc-a7d0-2998af266706","Type":"ContainerStarted","Data":"652ce7ff58bc4676118163af56f46e3da8d285982148ce0a3a7f8a9f39d9d12d"}
Sep 16 18:47:26.481500 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:26.481466    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2eb99315-c9d5-47fc-a7d0-2998af266706","Type":"ContainerStarted","Data":"e4d5976d850232cf561edf3a11b20ead5238b3aa05239034387f2d7366825fc5"}
Sep 16 18:47:26.481870 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:26.481536    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2eb99315-c9d5-47fc-a7d0-2998af266706","Type":"ContainerStarted","Data":"054e4f145f1d8b4747777e78bea7c5b0626d8e163f81aa6465b79077c3f18dc6"}
Sep 16 18:47:26.481870 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:26.481565    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2eb99315-c9d5-47fc-a7d0-2998af266706","Type":"ContainerStarted","Data":"4248f98abc2163ea869251f3a00eb632de2921f605d5ef699bbecf4220c315fc"}
Sep 16 18:47:26.481870 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:26.481593    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2eb99315-c9d5-47fc-a7d0-2998af266706","Type":"ContainerStarted","Data":"2c950fb44b29250b7c92c9469ecb6d75545d105054e9107c6b37b6249ea7493c"}
Sep 16 18:47:26.481870 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:26.481619    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2eb99315-c9d5-47fc-a7d0-2998af266706","Type":"ContainerStarted","Data":"a9c108ac299b63fe9bc67c7ef45d2342c0e3eb6fe837fb2b44c71927f348a17e"}
Sep 16 18:47:26.481870 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:26.481648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2eb99315-c9d5-47fc-a7d0-2998af266706","Type":"ContainerStarted","Data":"39045548cfd1690ea277fdb8529c07391aaff0914ebde74ba430328f45237b65"}
Sep 16 18:47:26.525217 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:26.525164    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.511138807 podStartE2EDuration="2.525148488s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.009681ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:26.520244907 +0000 UTC m=+474.337010520" watchObservedRunningTime="2026-09-16 18:47:26.525148488 +0000 UTC m=+474.341914074"
Sep 16 18:47:28.628670 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.628640    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:28.632527 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.632504    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.635284 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635214    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:47:28.635392 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635345    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f9cghajn0aeq1\""
Sep 16 18:47:28.635460 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635427    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:47:28.635521 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635460    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:47:28.635521 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635508    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:47:28.635629 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635577    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:47:28.635880 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635862    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-24zfq\""
Sep 16 18:47:28.635990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635896    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:47:28.635990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635918    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:47:28.635990 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.635954    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:47:28.636138 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.636058    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:47:28.636198 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.636178    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:47:28.636282 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.636185    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:47:28.636335 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.636189    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:47:28.639329 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.639309    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:47:28.642171 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.642149    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:47:28.651284 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.651260    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:28.791671 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791645    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.791671 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.791843 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791702    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.791843 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.791843 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.791932 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791856    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.791932 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.791932 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791890    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.791932 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.791932 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791924    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792078 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792078 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.791983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792078 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.792005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-config\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792078 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.792027    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b437de83-cad4-4a4e-ab8a-41866902bab8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792078 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.792049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b437de83-cad4-4a4e-ab8a-41866902bab8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.792097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.792117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.792131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdd7\" (UniqueName: \"kubernetes.io/projected/b437de83-cad4-4a4e-ab8a-41866902bab8-kube-api-access-qpdd7\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.792148    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.792226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.792169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893516 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893516 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893479    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893516 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893501    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893778 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893778 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893778 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893930 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893930 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893819    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893930 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.893930 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.893905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894278    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-config\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b437de83-cad4-4a4e-ab8a-41866902bab8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b437de83-cad4-4a4e-ab8a-41866902bab8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdd7\" (UniqueName: \"kubernetes.io/projected/b437de83-cad4-4a4e-ab8a-41866902bab8-kube-api-access-qpdd7\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.894949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.895506 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.895252    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.896081 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.895685    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.896081 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.895782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.896651 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.894382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.896651 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.896576    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.897964 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.896966    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.897964 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.897068    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.897964 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.897176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.897964 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.897378    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.897964 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.897448    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.897964 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.897517    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.898322 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.898084    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.898322 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.898115    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.898625 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.898558    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-config\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.898693 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.898626    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b437de83-cad4-4a4e-ab8a-41866902bab8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.898850 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.898829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b437de83-cad4-4a4e-ab8a-41866902bab8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.900140 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.900124    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.900192 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.900162    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b437de83-cad4-4a4e-ab8a-41866902bab8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.900575 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.900559    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b437de83-cad4-4a4e-ab8a-41866902bab8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.905986 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.905970    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdd7\" (UniqueName: \"kubernetes.io/projected/b437de83-cad4-4a4e-ab8a-41866902bab8-kube-api-access-qpdd7\") pod \"prometheus-k8s-0\" (UID: \"b437de83-cad4-4a4e-ab8a-41866902bab8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.942994 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:28.942974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:29.225293 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:29.225214    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:29.496304 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:29.496227    2575 generic.go:356] "Generic (PLEG): container finished" podID="b437de83-cad4-4a4e-ab8a-41866902bab8" containerID="6914cca1ca426ecb10da9e4bead90648703e9a14dbc8c612d06348a8ca027964" exitCode=0
Sep 16 18:47:29.496427 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:29.496318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b437de83-cad4-4a4e-ab8a-41866902bab8","Type":"ContainerDied","Data":"6914cca1ca426ecb10da9e4bead90648703e9a14dbc8c612d06348a8ca027964"}
Sep 16 18:47:29.496427 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:29.496386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b437de83-cad4-4a4e-ab8a-41866902bab8","Type":"ContainerStarted","Data":"be54767f4ababe372fe5dbdb1aaf987fd916bfdf4a50c023ca0d05580c4add3d"}
Sep 16 18:47:29.497510 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:29.497493    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:47:32.519330 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:32.519292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b437de83-cad4-4a4e-ab8a-41866902bab8","Type":"ContainerStarted","Data":"8f81daf824eb572a1ed21ece433a433de1710b057534e1225892f3fb34f2f26a"}
Sep 16 18:47:32.519702 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:32.519419    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b437de83-cad4-4a4e-ab8a-41866902bab8","Type":"ContainerStarted","Data":"e23e8a28f0cd3b55bcd4cb2c8754eac6aa2b8a88d3295aeff091148ad9aa2ed1"}
Sep 16 18:47:32.519702 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:32.519464    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b437de83-cad4-4a4e-ab8a-41866902bab8","Type":"ContainerStarted","Data":"bb15ce42b7f8ce720af37c4c73cdab40a1e5c2f0f7a30516d2c1f5f9c4da7bdc"}
Sep 16 18:47:33.526496 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:33.526462    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b437de83-cad4-4a4e-ab8a-41866902bab8","Type":"ContainerStarted","Data":"63efbba9b54a4b9310db03c98dcff551a17b274988e672c2d54bafbedd7ebf34"}
Sep 16 18:47:33.526835 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:33.526519    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b437de83-cad4-4a4e-ab8a-41866902bab8","Type":"ContainerStarted","Data":"69f788de585c3f8eefd4e65e0a18b8b57fcfe3ea81e985a1d8702047eea7c352"}
Sep 16 18:47:33.526835 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:33.526546    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b437de83-cad4-4a4e-ab8a-41866902bab8","Type":"ContainerStarted","Data":"819f1edf2ef3d44d4b2d9f0a3d8f85d291fcb42fadbcd51e0511a47949ed28d5"}
Sep 16 18:47:33.566826 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:33.566758    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7547047190000002 podStartE2EDuration="5.566738618s" totalImagesPullingTime="2.793459192s" totalInitContainerRuntime="18.574707ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:33.565226475 +0000 UTC m=+481.381992095" watchObservedRunningTime="2026-09-16 18:47:33.566738618 +0000 UTC m=+481.383504206"
Sep 16 18:47:33.943499 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:47:33.943461    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:28.943843 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:48:28.943753    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:28.959572 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:48:28.959546    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:29.765381 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:48:29.765351    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:50:24.040206 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.040172    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sqv45"]
Sep 16 18:50:24.043318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.043303    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:24.052666 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.052647    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:50:24.052789 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.052754    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:50:24.053575 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.053556    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6bqcv\""
Sep 16 18:50:24.066495 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.066468    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sqv45"]
Sep 16 18:50:24.122972 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.122939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qph\" (UniqueName: \"kubernetes.io/projected/5c6961ff-9c69-4df8-b68f-3943f95f1ecd-kube-api-access-m5qph\") pod \"cert-manager-webhook-6486ffd997-sqv45\" (UID: \"5c6961ff-9c69-4df8-b68f-3943f95f1ecd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:24.123116 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.122978    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c6961ff-9c69-4df8-b68f-3943f95f1ecd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sqv45\" (UID: \"5c6961ff-9c69-4df8-b68f-3943f95f1ecd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:24.224320 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.224287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qph\" (UniqueName: \"kubernetes.io/projected/5c6961ff-9c69-4df8-b68f-3943f95f1ecd-kube-api-access-m5qph\") pod \"cert-manager-webhook-6486ffd997-sqv45\" (UID: \"5c6961ff-9c69-4df8-b68f-3943f95f1ecd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:24.224505 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.224337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c6961ff-9c69-4df8-b68f-3943f95f1ecd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sqv45\" (UID: \"5c6961ff-9c69-4df8-b68f-3943f95f1ecd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:24.233382 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.233352    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c6961ff-9c69-4df8-b68f-3943f95f1ecd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sqv45\" (UID: \"5c6961ff-9c69-4df8-b68f-3943f95f1ecd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:24.235377 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.235356    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qph\" (UniqueName: \"kubernetes.io/projected/5c6961ff-9c69-4df8-b68f-3943f95f1ecd-kube-api-access-m5qph\") pod \"cert-manager-webhook-6486ffd997-sqv45\" (UID: \"5c6961ff-9c69-4df8-b68f-3943f95f1ecd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:24.361770 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.361747    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:24.637795 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:24.637696    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sqv45"]
Sep 16 18:50:24.640487 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:50:24.640453    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c6961ff_9c69_4df8_b68f_3943f95f1ecd.slice/crio-4f144598d1a9da23375ed00d963127f398ee8d306614a8e7b8c3db17a60e22db WatchSource:0}: Error finding container 4f144598d1a9da23375ed00d963127f398ee8d306614a8e7b8c3db17a60e22db: Status 404 returned error can't find the container with id 4f144598d1a9da23375ed00d963127f398ee8d306614a8e7b8c3db17a60e22db
Sep 16 18:50:25.206467 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:25.206432    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45" event={"ID":"5c6961ff-9c69-4df8-b68f-3943f95f1ecd","Type":"ContainerStarted","Data":"4f144598d1a9da23375ed00d963127f398ee8d306614a8e7b8c3db17a60e22db"}
Sep 16 18:50:28.222875 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:28.222845    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45" event={"ID":"5c6961ff-9c69-4df8-b68f-3943f95f1ecd","Type":"ContainerStarted","Data":"e55993c949a78275c5f65013b895f299e00700d82993e4072497f3232be6b991"}
Sep 16 18:50:28.223218 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:28.222975    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:28.242606 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:28.242567    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45" podStartSLOduration=1.098586157 podStartE2EDuration="4.242554575s" totalImagesPullingTime="3.143968418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:28.241634424 +0000 UTC m=+656.058400011" watchObservedRunningTime="2026-09-16 18:50:28.242554575 +0000 UTC m=+656.059320160"
Sep 16 18:50:34.229205 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:34.229172    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sqv45"
Sep 16 18:50:54.211354 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.211323    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"]
Sep 16 18:50:54.214822 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.214805    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.219118 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.219093    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 18:50:54.220407 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.220386    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gbmdk\""
Sep 16 18:50:54.220529 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.220455    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 18:50:54.220529 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.220511    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 18:50:54.220665 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.220589    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 18:50:54.230433 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.230412    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"]
Sep 16 18:50:54.273183 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.273161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b87e6211-2389-4142-9e81-5c03e111052c-manifests\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.273310 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.273188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62sc\" (UniqueName: \"kubernetes.io/projected/b87e6211-2389-4142-9e81-5c03e111052c-kube-api-access-x62sc\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.273310 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.273212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b87e6211-2389-4142-9e81-5c03e111052c-tmp\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.273310 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.273239    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b87e6211-2389-4142-9e81-5c03e111052c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.273310 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.273297    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b87e6211-2389-4142-9e81-5c03e111052c-webhook-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.374706 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.374678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b87e6211-2389-4142-9e81-5c03e111052c-manifests\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.374706 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.374719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62sc\" (UniqueName: \"kubernetes.io/projected/b87e6211-2389-4142-9e81-5c03e111052c-kube-api-access-x62sc\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.374892 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.374743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b87e6211-2389-4142-9e81-5c03e111052c-tmp\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.374892 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.374770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b87e6211-2389-4142-9e81-5c03e111052c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.375002 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.374930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b87e6211-2389-4142-9e81-5c03e111052c-webhook-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.375093 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.375072    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b87e6211-2389-4142-9e81-5c03e111052c-manifests\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.376987 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.376959    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b87e6211-2389-4142-9e81-5c03e111052c-tmp\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.377243 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.377225    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b87e6211-2389-4142-9e81-5c03e111052c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.377287 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.377259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b87e6211-2389-4142-9e81-5c03e111052c-webhook-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.389965 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.389940    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62sc\" (UniqueName: \"kubernetes.io/projected/b87e6211-2389-4142-9e81-5c03e111052c-kube-api-access-x62sc\") pod \"opendatahub-operator-controller-manager-85745479c7-zgzbf\" (UID: \"b87e6211-2389-4142-9e81-5c03e111052c\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.525132 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.525054    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:50:54.830520 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:54.830438    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"]
Sep 16 18:50:54.833745 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:50:54.833686    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87e6211_2389_4142_9e81_5c03e111052c.slice/crio-6ba157425df3b5455b7ac8cba32b1b48f799ac0de0031efbeddd1df4bddc57d8 WatchSource:0}: Error finding container 6ba157425df3b5455b7ac8cba32b1b48f799ac0de0031efbeddd1df4bddc57d8: Status 404 returned error can't find the container with id 6ba157425df3b5455b7ac8cba32b1b48f799ac0de0031efbeddd1df4bddc57d8
Sep 16 18:50:55.331903 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:55.331865    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf" event={"ID":"b87e6211-2389-4142-9e81-5c03e111052c","Type":"ContainerStarted","Data":"6ba157425df3b5455b7ac8cba32b1b48f799ac0de0031efbeddd1df4bddc57d8"}
Sep 16 18:50:58.348420 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:58.348385    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf" event={"ID":"b87e6211-2389-4142-9e81-5c03e111052c","Type":"ContainerStarted","Data":"28805c63c27b85819cfb4bd305ff8ad338a5f38fccca7b7f5e3e6d538cddc4cd"}
Sep 16 18:50:59.354750 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:59.354700    2575 generic.go:356] "Generic (PLEG): container finished" podID="b87e6211-2389-4142-9e81-5c03e111052c" containerID="28805c63c27b85819cfb4bd305ff8ad338a5f38fccca7b7f5e3e6d538cddc4cd" exitCode=0
Sep 16 18:50:59.355104 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:50:59.354763    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf" event={"ID":"b87e6211-2389-4142-9e81-5c03e111052c","Type":"ContainerDied","Data":"28805c63c27b85819cfb4bd305ff8ad338a5f38fccca7b7f5e3e6d538cddc4cd"}
Sep 16 18:51:00.361944 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:00.361908    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf" event={"ID":"b87e6211-2389-4142-9e81-5c03e111052c","Type":"ContainerStarted","Data":"7ad15494dd779e51c30bea473b69c448b2f359b5bce8f874e843dccb73b8b531"}
Sep 16 18:51:00.362343 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:00.362034    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:51:00.392997 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:00.392945    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf" podStartSLOduration=1.951273358 podStartE2EDuration="6.392921668s" totalImagesPullingTime="3.410682484s" totalInitContainerRuntime="1.030965826s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:00.392905722 +0000 UTC m=+688.209671310" watchObservedRunningTime="2026-09-16 18:51:00.392921668 +0000 UTC m=+688.209687255"
Sep 16 18:51:11.371347 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:11.371311    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-zgzbf"
Sep 16 18:51:33.798535 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.798461    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"]
Sep 16 18:51:33.804919 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.804900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:33.808608 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.808580    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:51:33.809683 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.809663    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:51:33.809683 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.809676    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:51:33.809846 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.809676    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:51:33.809846 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.809748    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:51:33.809846 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.809826    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z74zj\""
Sep 16 18:51:33.815735 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.815697    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"]
Sep 16 18:51:33.919320 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.919296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55ee4c05-e435-4937-a9d9-5d5effb53a35-cert\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:33.919444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.919336    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55ee4c05-e435-4937-a9d9-5d5effb53a35-metrics-cert\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:33.919444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.919393    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwk4x\" (UniqueName: \"kubernetes.io/projected/55ee4c05-e435-4937-a9d9-5d5effb53a35-kube-api-access-rwk4x\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:33.919444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:33.919437    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/55ee4c05-e435-4937-a9d9-5d5effb53a35-manager-config\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.020693 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.020665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwk4x\" (UniqueName: \"kubernetes.io/projected/55ee4c05-e435-4937-a9d9-5d5effb53a35-kube-api-access-rwk4x\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.020820 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.020725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/55ee4c05-e435-4937-a9d9-5d5effb53a35-manager-config\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.020820 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.020778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55ee4c05-e435-4937-a9d9-5d5effb53a35-cert\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.020820 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.020810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55ee4c05-e435-4937-a9d9-5d5effb53a35-metrics-cert\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.021504 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.021484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/55ee4c05-e435-4937-a9d9-5d5effb53a35-manager-config\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.023226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.023202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55ee4c05-e435-4937-a9d9-5d5effb53a35-metrics-cert\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.023306 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.023208    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55ee4c05-e435-4937-a9d9-5d5effb53a35-cert\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.045351 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.045332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwk4x\" (UniqueName: \"kubernetes.io/projected/55ee4c05-e435-4937-a9d9-5d5effb53a35-kube-api-access-rwk4x\") pod \"lws-controller-manager-5f86d45999-2br8t\" (UID: \"55ee4c05-e435-4937-a9d9-5d5effb53a35\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.114346 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.114323    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:34.406525 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.406495    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"]
Sep 16 18:51:34.408275 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:51:34.408246    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ee4c05_e435_4937_a9d9_5d5effb53a35.slice/crio-43b4f3d654803b6479cdbf4f1713ef7b7f5979889ba706364adc2b6f01db72b3 WatchSource:0}: Error finding container 43b4f3d654803b6479cdbf4f1713ef7b7f5979889ba706364adc2b6f01db72b3: Status 404 returned error can't find the container with id 43b4f3d654803b6479cdbf4f1713ef7b7f5979889ba706364adc2b6f01db72b3
Sep 16 18:51:34.519256 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:34.519210    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t" event={"ID":"55ee4c05-e435-4937-a9d9-5d5effb53a35","Type":"ContainerStarted","Data":"43b4f3d654803b6479cdbf4f1713ef7b7f5979889ba706364adc2b6f01db72b3"}
Sep 16 18:51:36.534198 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.534166    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7996r"]
Sep 16 18:51:36.537154 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.537132    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:36.539853 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.539828    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 18:51:36.540051 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.540037    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-27nr4\""
Sep 16 18:51:36.541365 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.541336    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lt5v\" (UniqueName: \"kubernetes.io/projected/79856847-afeb-4b2c-aaff-5b1a103b2a7f-kube-api-access-9lt5v\") pod \"kserve-controller-manager-d6f4b65b8-7996r\" (UID: \"79856847-afeb-4b2c-aaff-5b1a103b2a7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:36.541455 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.541417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79856847-afeb-4b2c-aaff-5b1a103b2a7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-7996r\" (UID: \"79856847-afeb-4b2c-aaff-5b1a103b2a7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:36.551872 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.551848    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7996r"]
Sep 16 18:51:36.573437 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.573404    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"]
Sep 16 18:51:36.576367 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.576350    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:36.579981 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.579955    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:51:36.580226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.580205    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gtg92\""
Sep 16 18:51:36.585833 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.585812    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"]
Sep 16 18:51:36.641975 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.641939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lt5v\" (UniqueName: \"kubernetes.io/projected/79856847-afeb-4b2c-aaff-5b1a103b2a7f-kube-api-access-9lt5v\") pod \"kserve-controller-manager-d6f4b65b8-7996r\" (UID: \"79856847-afeb-4b2c-aaff-5b1a103b2a7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:36.642174 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.642000    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3c84cb-6463-40ba-a1f0-355e9b6e132e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9nmll\" (UID: \"4a3c84cb-6463-40ba-a1f0-355e9b6e132e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:36.642174 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.642121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79856847-afeb-4b2c-aaff-5b1a103b2a7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-7996r\" (UID: \"79856847-afeb-4b2c-aaff-5b1a103b2a7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:36.642286 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.642174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddhkq\" (UniqueName: \"kubernetes.io/projected/4a3c84cb-6463-40ba-a1f0-355e9b6e132e-kube-api-access-ddhkq\") pod \"llmisvc-controller-manager-777c6bd7d5-9nmll\" (UID: \"4a3c84cb-6463-40ba-a1f0-355e9b6e132e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:36.642286 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:51:36.642228    2575 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 18:51:36.642382 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:51:36.642325    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79856847-afeb-4b2c-aaff-5b1a103b2a7f-cert podName:79856847-afeb-4b2c-aaff-5b1a103b2a7f nodeName:}" failed. No retries permitted until 2026-09-16 18:51:37.142304646 +0000 UTC m=+724.959070210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/79856847-afeb-4b2c-aaff-5b1a103b2a7f-cert") pod "kserve-controller-manager-d6f4b65b8-7996r" (UID: "79856847-afeb-4b2c-aaff-5b1a103b2a7f") : secret "kserve-webhook-server-cert" not found
Sep 16 18:51:36.660503 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.660470    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lt5v\" (UniqueName: \"kubernetes.io/projected/79856847-afeb-4b2c-aaff-5b1a103b2a7f-kube-api-access-9lt5v\") pod \"kserve-controller-manager-d6f4b65b8-7996r\" (UID: \"79856847-afeb-4b2c-aaff-5b1a103b2a7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:36.742776 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.742737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhkq\" (UniqueName: \"kubernetes.io/projected/4a3c84cb-6463-40ba-a1f0-355e9b6e132e-kube-api-access-ddhkq\") pod \"llmisvc-controller-manager-777c6bd7d5-9nmll\" (UID: \"4a3c84cb-6463-40ba-a1f0-355e9b6e132e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:36.742961 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.742803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3c84cb-6463-40ba-a1f0-355e9b6e132e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9nmll\" (UID: \"4a3c84cb-6463-40ba-a1f0-355e9b6e132e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:36.742961 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:51:36.742919    2575 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 18:51:36.743074 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:51:36.742975    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a3c84cb-6463-40ba-a1f0-355e9b6e132e-cert podName:4a3c84cb-6463-40ba-a1f0-355e9b6e132e nodeName:}" failed. No retries permitted until 2026-09-16 18:51:37.242959066 +0000 UTC m=+725.059724632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4a3c84cb-6463-40ba-a1f0-355e9b6e132e-cert") pod "llmisvc-controller-manager-777c6bd7d5-9nmll" (UID: "4a3c84cb-6463-40ba-a1f0-355e9b6e132e") : secret "llmisvc-webhook-server-cert" not found
Sep 16 18:51:36.776516 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:36.776485    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhkq\" (UniqueName: \"kubernetes.io/projected/4a3c84cb-6463-40ba-a1f0-355e9b6e132e-kube-api-access-ddhkq\") pod \"llmisvc-controller-manager-777c6bd7d5-9nmll\" (UID: \"4a3c84cb-6463-40ba-a1f0-355e9b6e132e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:37.146009 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:37.145975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79856847-afeb-4b2c-aaff-5b1a103b2a7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-7996r\" (UID: \"79856847-afeb-4b2c-aaff-5b1a103b2a7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:37.148372 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:37.148340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79856847-afeb-4b2c-aaff-5b1a103b2a7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-7996r\" (UID: \"79856847-afeb-4b2c-aaff-5b1a103b2a7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:37.246976 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:37.246941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3c84cb-6463-40ba-a1f0-355e9b6e132e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9nmll\" (UID: \"4a3c84cb-6463-40ba-a1f0-355e9b6e132e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:37.249247 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:37.249219    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a3c84cb-6463-40ba-a1f0-355e9b6e132e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9nmll\" (UID: \"4a3c84cb-6463-40ba-a1f0-355e9b6e132e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:37.447289 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:37.447209    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:37.487539 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:37.487502    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:37.769963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:37.769924    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7996r"]
Sep 16 18:51:37.774794 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:51:37.774694    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79856847_afeb_4b2c_aaff_5b1a103b2a7f.slice/crio-d4e662268e060638487248353622ba7b383e1723c5cbb84351da5c7b9a14c838 WatchSource:0}: Error finding container d4e662268e060638487248353622ba7b383e1723c5cbb84351da5c7b9a14c838: Status 404 returned error can't find the container with id d4e662268e060638487248353622ba7b383e1723c5cbb84351da5c7b9a14c838
Sep 16 18:51:37.785441 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:37.785419    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"]
Sep 16 18:51:37.787770 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:51:37.787746    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4a3c84cb_6463_40ba_a1f0_355e9b6e132e.slice/crio-51c6dbe15284fd5e88f47914208ff0ef27c36e75a04c07d07d73efaf94eaec24 WatchSource:0}: Error finding container 51c6dbe15284fd5e88f47914208ff0ef27c36e75a04c07d07d73efaf94eaec24: Status 404 returned error can't find the container with id 51c6dbe15284fd5e88f47914208ff0ef27c36e75a04c07d07d73efaf94eaec24
Sep 16 18:51:38.552525 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:38.552432    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll" event={"ID":"4a3c84cb-6463-40ba-a1f0-355e9b6e132e","Type":"ContainerStarted","Data":"51c6dbe15284fd5e88f47914208ff0ef27c36e75a04c07d07d73efaf94eaec24"}
Sep 16 18:51:38.555087 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:38.555015    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r" event={"ID":"79856847-afeb-4b2c-aaff-5b1a103b2a7f","Type":"ContainerStarted","Data":"d4e662268e060638487248353622ba7b383e1723c5cbb84351da5c7b9a14c838"}
Sep 16 18:51:42.581367 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:42.581327    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r" event={"ID":"79856847-afeb-4b2c-aaff-5b1a103b2a7f","Type":"ContainerStarted","Data":"f4b9f21d6705019b20b9ef7887a4921b139327359865fd352bd87a5beb9a725a"}
Sep 16 18:51:42.581842 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:42.581404    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:51:42.582833 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:42.582806    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t" event={"ID":"55ee4c05-e435-4937-a9d9-5d5effb53a35","Type":"ContainerStarted","Data":"2d84b1189b7396c13d80ca79d9d5de24c048d2a759ed454b23397a4ecdd42df2"}
Sep 16 18:51:42.582956 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:42.582928    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:51:42.613010 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:42.612922    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r" podStartSLOduration=2.515282416 podStartE2EDuration="6.612900791s" totalImagesPullingTime="4.097618375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:42.612436504 +0000 UTC m=+730.429202097" watchObservedRunningTime="2026-09-16 18:51:42.612900791 +0000 UTC m=+730.429666378"
Sep 16 18:51:42.706363 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:42.706250    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t" podStartSLOduration=2.285205309 podStartE2EDuration="9.706229467s" totalImagesPullingTime="7.421024158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:42.706223435 +0000 UTC m=+730.522989022" watchObservedRunningTime="2026-09-16 18:51:42.706229467 +0000 UTC m=+730.522995054"
Sep 16 18:51:43.588370 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:43.588334    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll" event={"ID":"4a3c84cb-6463-40ba-a1f0-355e9b6e132e","Type":"ContainerStarted","Data":"b7c77bdde0eba0fbe1a3e627ae32ff29f823d7516ee49a54d5528d58cc70d04d"}
Sep 16 18:51:43.588817 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:43.588420    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:51:43.610764 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:43.610718    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll" podStartSLOduration=2.895159199 podStartE2EDuration="7.610690208s" totalImagesPullingTime="4.715531009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:43.60860972 +0000 UTC m=+731.425375307" watchObservedRunningTime="2026-09-16 18:51:43.610690208 +0000 UTC m=+731.427455808"
Sep 16 18:51:53.590752 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:51:53.590697    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-2br8t"
Sep 16 18:52:13.593166 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:13.593134    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7996r"
Sep 16 18:52:14.600797 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:14.600762    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9nmll"
Sep 16 18:52:43.178815 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.178776    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cf59656f5-4vbd9"]
Sep 16 18:52:43.181099 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.181082    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.183391 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.183366    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:52:43.184436 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.184395    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rwxg4\""
Sep 16 18:52:43.184436 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.184398    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:52:43.184605 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.184495    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:52:43.184605 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.184502    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:52:43.184605 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.184510    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:52:43.184774 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.184758    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:52:43.184896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.184880    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:52:43.190127 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.190107    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:52:43.199297 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.199279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-console-oauth-config\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.199373 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.199305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-service-ca\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.199373 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.199355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-console-serving-cert\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.199446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.199379    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-trusted-ca-bundle\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.199446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.199394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-console-config\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.199446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.199411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-oauth-serving-cert\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.199446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.199435    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f92js\" (UniqueName: \"kubernetes.io/projected/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-kube-api-access-f92js\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.232668 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.232637    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf59656f5-4vbd9"]
Sep 16 18:52:43.300560 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.300521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-console-oauth-config\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.300560 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.300557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-service-ca\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.300785 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.300595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-console-serving-cert\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.300785 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.300704    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-trusted-ca-bundle\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.300785 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.300773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-console-config\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.300956 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.300934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-oauth-serving-cert\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.301013 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.300984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f92js\" (UniqueName: \"kubernetes.io/projected/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-kube-api-access-f92js\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.301444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.301419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-console-config\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.301560 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.301419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-service-ca\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.301631 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.301591    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-oauth-serving-cert\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.301811 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.301792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-trusted-ca-bundle\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.302973 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.302955    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-console-oauth-config\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.303080 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.303064    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-console-serving-cert\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.309944 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.309920    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f92js\" (UniqueName: \"kubernetes.io/projected/d53498b9-0cb2-45fb-9c1b-4e49aa13b21d-kube-api-access-f92js\") pod \"console-6cf59656f5-4vbd9\" (UID: \"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d\") " pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.490565 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.490466    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:43.763631 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.763604    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf59656f5-4vbd9"]
Sep 16 18:52:43.766271 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:52:43.766242    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53498b9_0cb2_45fb_9c1b_4e49aa13b21d.slice/crio-c82f4732c35367466263d60d9fadba28cf172c5e7d121ef4cbf6aa3c126b9fbf WatchSource:0}: Error finding container c82f4732c35367466263d60d9fadba28cf172c5e7d121ef4cbf6aa3c126b9fbf: Status 404 returned error can't find the container with id c82f4732c35367466263d60d9fadba28cf172c5e7d121ef4cbf6aa3c126b9fbf
Sep 16 18:52:43.871276 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.871238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf59656f5-4vbd9" event={"ID":"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d","Type":"ContainerStarted","Data":"7f1a97307a5686be0af24fbb7ed775310cd9a54cb5adac412ca57b8cd487e114"}
Sep 16 18:52:43.871450 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.871287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf59656f5-4vbd9" event={"ID":"d53498b9-0cb2-45fb-9c1b-4e49aa13b21d","Type":"ContainerStarted","Data":"c82f4732c35367466263d60d9fadba28cf172c5e7d121ef4cbf6aa3c126b9fbf"}
Sep 16 18:52:43.893974 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:43.893919    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cf59656f5-4vbd9" podStartSLOduration=0.893903756 podStartE2EDuration="893.903756ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:43.891422663 +0000 UTC m=+791.708188253" watchObservedRunningTime="2026-09-16 18:52:43.893903756 +0000 UTC m=+791.710669341"
Sep 16 18:52:50.381984 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.381943    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"]
Sep 16 18:52:50.384660 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.384631    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:52:50.388515 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.388494    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:52:50.388650 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.388627    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:52:50.388748 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.388733    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8pkfh\""
Sep 16 18:52:50.412400 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.412369    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"]
Sep 16 18:52:50.465361 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.465320    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czlqb\" (UniqueName: \"kubernetes.io/projected/25e8cba4-f72e-42fa-af20-ad059b34fccf-kube-api-access-czlqb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" (UID: \"25e8cba4-f72e-42fa-af20-ad059b34fccf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:52:50.465538 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.465385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25e8cba4-f72e-42fa-af20-ad059b34fccf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" (UID: \"25e8cba4-f72e-42fa-af20-ad059b34fccf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:52:50.566575 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.566534    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czlqb\" (UniqueName: \"kubernetes.io/projected/25e8cba4-f72e-42fa-af20-ad059b34fccf-kube-api-access-czlqb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" (UID: \"25e8cba4-f72e-42fa-af20-ad059b34fccf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:52:50.566765 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.566596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25e8cba4-f72e-42fa-af20-ad059b34fccf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" (UID: \"25e8cba4-f72e-42fa-af20-ad059b34fccf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:52:50.566971 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.566955    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25e8cba4-f72e-42fa-af20-ad059b34fccf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" (UID: \"25e8cba4-f72e-42fa-af20-ad059b34fccf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:52:50.575614 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.575584    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czlqb\" (UniqueName: \"kubernetes.io/projected/25e8cba4-f72e-42fa-af20-ad059b34fccf-kube-api-access-czlqb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" (UID: \"25e8cba4-f72e-42fa-af20-ad059b34fccf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:52:50.694942 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.694852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:52:50.990006 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.989979    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"]
Sep 16 18:52:50.992621 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:52:50.992591    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e8cba4_f72e_42fa_af20_ad059b34fccf.slice/crio-ec59dc57eef87e253079d6717a5854a2d656eabaa2011da1b5797dc338278641 WatchSource:0}: Error finding container ec59dc57eef87e253079d6717a5854a2d656eabaa2011da1b5797dc338278641: Status 404 returned error can't find the container with id ec59dc57eef87e253079d6717a5854a2d656eabaa2011da1b5797dc338278641
Sep 16 18:52:50.995399 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:50.995376    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:52:51.911821 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:51.911778    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r" event={"ID":"25e8cba4-f72e-42fa-af20-ad059b34fccf","Type":"ContainerStarted","Data":"ec59dc57eef87e253079d6717a5854a2d656eabaa2011da1b5797dc338278641"}
Sep 16 18:52:53.491090 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:53.491056    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:53.491515 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:53.491121    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:53.496828 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:53.496798    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:53.927440 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:53.927410    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cf59656f5-4vbd9"
Sep 16 18:52:54.928568 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:54.928535    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r" event={"ID":"25e8cba4-f72e-42fa-af20-ad059b34fccf","Type":"ContainerStarted","Data":"038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a"}
Sep 16 18:52:54.928995 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:54.928793    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:52:54.957616 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:54.957566    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r" podStartSLOduration=1.122931725 podStartE2EDuration="4.95755359s" totalImagesPullingTime="3.834621865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:54.954885566 +0000 UTC m=+802.771651152" watchObservedRunningTime="2026-09-16 18:52:54.95755359 +0000 UTC m=+802.774319175"
Sep 16 18:52:55.025070 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.025044    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"]
Sep 16 18:52:55.028327 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.028309    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"
Sep 16 18:52:55.031335 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.031314    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 18:52:55.031621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.031604    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wfnrg\""
Sep 16 18:52:55.042752 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.042730    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"]
Sep 16 18:52:55.112624 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.112596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk98t\" (UniqueName: \"kubernetes.io/projected/c30e1190-8aec-49ac-bbf7-77e51d7602ba-kube-api-access-xk98t\") pod \"dns-operator-controller-manager-7b84cd7c9c-rhgjj\" (UID: \"c30e1190-8aec-49ac-bbf7-77e51d7602ba\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"
Sep 16 18:52:55.214051 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.214026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk98t\" (UniqueName: \"kubernetes.io/projected/c30e1190-8aec-49ac-bbf7-77e51d7602ba-kube-api-access-xk98t\") pod \"dns-operator-controller-manager-7b84cd7c9c-rhgjj\" (UID: \"c30e1190-8aec-49ac-bbf7-77e51d7602ba\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"
Sep 16 18:52:55.226701 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.226677    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk98t\" (UniqueName: \"kubernetes.io/projected/c30e1190-8aec-49ac-bbf7-77e51d7602ba-kube-api-access-xk98t\") pod \"dns-operator-controller-manager-7b84cd7c9c-rhgjj\" (UID: \"c30e1190-8aec-49ac-bbf7-77e51d7602ba\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"
Sep 16 18:52:55.338156 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.338115    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"
Sep 16 18:52:55.623725 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.623673    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"]
Sep 16 18:52:55.627470 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:52:55.627434    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30e1190_8aec_49ac_bbf7_77e51d7602ba.slice/crio-a037deda76921c583161451330701cbe106ee81ac66949eeba7fce974253dc04 WatchSource:0}: Error finding container a037deda76921c583161451330701cbe106ee81ac66949eeba7fce974253dc04: Status 404 returned error can't find the container with id a037deda76921c583161451330701cbe106ee81ac66949eeba7fce974253dc04
Sep 16 18:52:55.935025 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:55.934991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj" event={"ID":"c30e1190-8aec-49ac-bbf7-77e51d7602ba","Type":"ContainerStarted","Data":"a037deda76921c583161451330701cbe106ee81ac66949eeba7fce974253dc04"}
Sep 16 18:52:58.952763 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:58.952731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj" event={"ID":"c30e1190-8aec-49ac-bbf7-77e51d7602ba","Type":"ContainerStarted","Data":"c08a5f9b38b8ee3c915ee4a630bbf592315477c3f2e63e2be08ccb66fc4835f1"}
Sep 16 18:52:58.953151 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:58.952847    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"
Sep 16 18:52:58.975234 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:52:58.975192    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj" podStartSLOduration=1.5239442460000001 podStartE2EDuration="3.975151996s" totalImagesPullingTime="2.45120775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:58.974185924 +0000 UTC m=+806.790951535" watchObservedRunningTime="2026-09-16 18:52:58.975151996 +0000 UTC m=+806.791917581"
Sep 16 18:53:05.937775 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:05.937741    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:53:08.059144 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.059110    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"]
Sep 16 18:53:08.059521 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.059309    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r" podUID="25e8cba4-f72e-42fa-af20-ad059b34fccf" containerName="manager" containerID="cri-o://038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a" gracePeriod=2
Sep 16 18:53:08.090365 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.090336    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"]
Sep 16 18:53:08.097055 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.097028    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"]
Sep 16 18:53:08.097402 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.097390    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e8cba4-f72e-42fa-af20-ad059b34fccf" containerName="manager"
Sep 16 18:53:08.097440 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.097404    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e8cba4-f72e-42fa-af20-ad059b34fccf" containerName="manager"
Sep 16 18:53:08.097476 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.097458    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e8cba4-f72e-42fa-af20-ad059b34fccf" containerName="manager"
Sep 16 18:53:08.100244 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.100229    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:08.109517 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:53:08.109486    2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"25e8cba4-f72e-42fa-af20-ad059b34fccf\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\": pods \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:53:08.145044 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.145011    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"]
Sep 16 18:53:08.161386 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:53:08.161340    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="25e8cba4-f72e-42fa-af20-ad059b34fccf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:53:08.162362 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.162341    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"]
Sep 16 18:53:08.166083 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.166057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"
Sep 16 18:53:08.169763 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:53:08.169736    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="25e8cba4-f72e-42fa-af20-ad059b34fccf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:53:08.169927 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.169906    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rxc4d\""
Sep 16 18:53:08.180022 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.179999    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"]
Sep 16 18:53:08.237248 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.237104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/077d1934-8deb-4b4c-8062-7bbd3a121ea7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kv4q\" (UID: \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:08.237248 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.237235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd27q\" (UniqueName: \"kubernetes.io/projected/077d1934-8deb-4b4c-8062-7bbd3a121ea7-kube-api-access-jd27q\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kv4q\" (UID: \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:08.303965 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.303942    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:53:08.306213 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:53:08.306187    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-tfc6r\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="25e8cba4-f72e-42fa-af20-ad059b34fccf" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:53:08.338607 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.338580    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd27q\" (UniqueName: \"kubernetes.io/projected/077d1934-8deb-4b4c-8062-7bbd3a121ea7-kube-api-access-jd27q\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kv4q\" (UID: \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:08.338727 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.338617    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5n5\" (UniqueName: \"kubernetes.io/projected/a4c325d0-b372-46e0-8138-7d181cf16ab7-kube-api-access-fm5n5\") pod \"limitador-operator-controller-manager-74495c69f8-q4z47\" (UID: \"a4c325d0-b372-46e0-8138-7d181cf16ab7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"
Sep 16 18:53:08.338727 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.338658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/077d1934-8deb-4b4c-8062-7bbd3a121ea7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kv4q\" (UID: \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:08.339036 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.339020    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/077d1934-8deb-4b4c-8062-7bbd3a121ea7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kv4q\" (UID: \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:08.350859 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.350835    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd27q\" (UniqueName: \"kubernetes.io/projected/077d1934-8deb-4b4c-8062-7bbd3a121ea7-kube-api-access-jd27q\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9kv4q\" (UID: \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:08.439001 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.438960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:08.439158 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.439054    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25e8cba4-f72e-42fa-af20-ad059b34fccf-extensions-socket-volume\") pod \"25e8cba4-f72e-42fa-af20-ad059b34fccf\" (UID: \"25e8cba4-f72e-42fa-af20-ad059b34fccf\") "
Sep 16 18:53:08.439158 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.439091    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czlqb\" (UniqueName: \"kubernetes.io/projected/25e8cba4-f72e-42fa-af20-ad059b34fccf-kube-api-access-czlqb\") pod \"25e8cba4-f72e-42fa-af20-ad059b34fccf\" (UID: \"25e8cba4-f72e-42fa-af20-ad059b34fccf\") "
Sep 16 18:53:08.439374 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.439353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5n5\" (UniqueName: \"kubernetes.io/projected/a4c325d0-b372-46e0-8138-7d181cf16ab7-kube-api-access-fm5n5\") pod \"limitador-operator-controller-manager-74495c69f8-q4z47\" (UID: \"a4c325d0-b372-46e0-8138-7d181cf16ab7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"
Sep 16 18:53:08.439603 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.439583    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e8cba4-f72e-42fa-af20-ad059b34fccf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "25e8cba4-f72e-42fa-af20-ad059b34fccf" (UID: "25e8cba4-f72e-42fa-af20-ad059b34fccf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:08.441429 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.441401    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e8cba4-f72e-42fa-af20-ad059b34fccf-kube-api-access-czlqb" (OuterVolumeSpecName: "kube-api-access-czlqb") pod "25e8cba4-f72e-42fa-af20-ad059b34fccf" (UID: "25e8cba4-f72e-42fa-af20-ad059b34fccf"). InnerVolumeSpecName "kube-api-access-czlqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:08.451170 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.451150    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5n5\" (UniqueName: \"kubernetes.io/projected/a4c325d0-b372-46e0-8138-7d181cf16ab7-kube-api-access-fm5n5\") pod \"limitador-operator-controller-manager-74495c69f8-q4z47\" (UID: \"a4c325d0-b372-46e0-8138-7d181cf16ab7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"
Sep 16 18:53:08.479696 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.479651    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"
Sep 16 18:53:08.540903 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.540630    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/25e8cba4-f72e-42fa-af20-ad059b34fccf-extensions-socket-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:53:08.540903 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.540662    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czlqb\" (UniqueName: \"kubernetes.io/projected/25e8cba4-f72e-42fa-af20-ad059b34fccf-kube-api-access-czlqb\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:53:08.620596 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.620524    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25e8cba4-f72e-42fa-af20-ad059b34fccf" path="/var/lib/kubelet/pods/25e8cba4-f72e-42fa-af20-ad059b34fccf/volumes"
Sep 16 18:53:08.723075 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.723040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"]
Sep 16 18:53:08.727909 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:53:08.727882    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077d1934_8deb_4b4c_8062_7bbd3a121ea7.slice/crio-d357bfee7c2680bdf31c2790264beb8cc1d97aa9e1b13f203d7fad0c263db71f WatchSource:0}: Error finding container d357bfee7c2680bdf31c2790264beb8cc1d97aa9e1b13f203d7fad0c263db71f: Status 404 returned error can't find the container with id d357bfee7c2680bdf31c2790264beb8cc1d97aa9e1b13f203d7fad0c263db71f
Sep 16 18:53:08.757489 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:08.757464    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"]
Sep 16 18:53:08.766824 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:53:08.766791    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c325d0_b372_46e0_8138_7d181cf16ab7.slice/crio-a95a22c2fd373ab3364854b791058e1792fb50f5c10bbea20cef55a88f76c07d WatchSource:0}: Error finding container a95a22c2fd373ab3364854b791058e1792fb50f5c10bbea20cef55a88f76c07d: Status 404 returned error can't find the container with id a95a22c2fd373ab3364854b791058e1792fb50f5c10bbea20cef55a88f76c07d
Sep 16 18:53:09.001992 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.001894    2575 generic.go:356] "Generic (PLEG): container finished" podID="25e8cba4-f72e-42fa-af20-ad059b34fccf" containerID="038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a" exitCode=0
Sep 16 18:53:09.001992 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.001942    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tfc6r"
Sep 16 18:53:09.002216 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.001998    2575 scope.go:117] "RemoveContainer" containerID="038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a"
Sep 16 18:53:09.004128 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.004037    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q" event={"ID":"077d1934-8deb-4b4c-8062-7bbd3a121ea7","Type":"ContainerStarted","Data":"89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4"}
Sep 16 18:53:09.004128 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.004108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q" event={"ID":"077d1934-8deb-4b4c-8062-7bbd3a121ea7","Type":"ContainerStarted","Data":"d357bfee7c2680bdf31c2790264beb8cc1d97aa9e1b13f203d7fad0c263db71f"}
Sep 16 18:53:09.004322 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.004143    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:09.005601 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.005580    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47" event={"ID":"a4c325d0-b372-46e0-8138-7d181cf16ab7","Type":"ContainerStarted","Data":"a95a22c2fd373ab3364854b791058e1792fb50f5c10bbea20cef55a88f76c07d"}
Sep 16 18:53:09.010936 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.010917    2575 scope.go:117] "RemoveContainer" containerID="038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a"
Sep 16 18:53:09.011234 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:53:09.011216    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a\": container with ID starting with 038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a not found: ID does not exist" containerID="038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a"
Sep 16 18:53:09.011283 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.011242    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a"} err="failed to get container status \"038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a\": rpc error: code = NotFound desc = could not find container \"038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a\": container with ID starting with 038955c615bf7977e895cf78ff1b7763bb66183658fb7616ed9e9238ed96ac2a not found: ID does not exist"
Sep 16 18:53:09.035486 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.035429    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q" podStartSLOduration=1.035411631 podStartE2EDuration="1.035411631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:09.033309765 +0000 UTC m=+816.850075352" watchObservedRunningTime="2026-09-16 18:53:09.035411631 +0000 UTC m=+816.852177216"
Sep 16 18:53:09.961219 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:09.961188    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rhgjj"
Sep 16 18:53:11.020970 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:11.020880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47" event={"ID":"a4c325d0-b372-46e0-8138-7d181cf16ab7","Type":"ContainerStarted","Data":"3ed7f27c9e6fa96435580fb2387b3d058b6a939ff882d49f80168934f6d5973f"}
Sep 16 18:53:11.021348 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:11.021036    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"
Sep 16 18:53:11.045839 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:11.045782    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47" podStartSLOduration=1.094427559 podStartE2EDuration="3.045763671s" totalImagesPullingTime="1.951336112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:11.044198578 +0000 UTC m=+818.860964164" watchObservedRunningTime="2026-09-16 18:53:11.045763671 +0000 UTC m=+818.862529259"
Sep 16 18:53:20.015382 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:20.015350    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:22.028036 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:22.028005    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q4z47"
Sep 16 18:53:27.448760 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:27.448728    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"]
Sep 16 18:53:27.449213 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:27.448944    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q" podUID="077d1934-8deb-4b4c-8062-7bbd3a121ea7" containerName="manager" containerID="cri-o://89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4" gracePeriod=10
Sep 16 18:53:27.697040 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:27.697015    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:27.819684 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:27.819585    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/077d1934-8deb-4b4c-8062-7bbd3a121ea7-extensions-socket-volume\") pod \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\" (UID: \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\") "
Sep 16 18:53:27.819684 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:27.819645    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd27q\" (UniqueName: \"kubernetes.io/projected/077d1934-8deb-4b4c-8062-7bbd3a121ea7-kube-api-access-jd27q\") pod \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\" (UID: \"077d1934-8deb-4b4c-8062-7bbd3a121ea7\") "
Sep 16 18:53:27.820082 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:27.820052    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077d1934-8deb-4b4c-8062-7bbd3a121ea7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "077d1934-8deb-4b4c-8062-7bbd3a121ea7" (UID: "077d1934-8deb-4b4c-8062-7bbd3a121ea7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:27.821804 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:27.821780    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077d1934-8deb-4b4c-8062-7bbd3a121ea7-kube-api-access-jd27q" (OuterVolumeSpecName: "kube-api-access-jd27q") pod "077d1934-8deb-4b4c-8062-7bbd3a121ea7" (UID: "077d1934-8deb-4b4c-8062-7bbd3a121ea7"). InnerVolumeSpecName "kube-api-access-jd27q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:27.921175 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:27.921142    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/077d1934-8deb-4b4c-8062-7bbd3a121ea7-extensions-socket-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:53:27.921175 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:27.921172    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd27q\" (UniqueName: \"kubernetes.io/projected/077d1934-8deb-4b4c-8062-7bbd3a121ea7-kube-api-access-jd27q\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:53:28.104318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.104286    2575 generic.go:356] "Generic (PLEG): container finished" podID="077d1934-8deb-4b4c-8062-7bbd3a121ea7" containerID="89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4" exitCode=0
Sep 16 18:53:28.104479 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.104331    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"
Sep 16 18:53:28.104479 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.104370    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q" event={"ID":"077d1934-8deb-4b4c-8062-7bbd3a121ea7","Type":"ContainerDied","Data":"89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4"}
Sep 16 18:53:28.104479 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.104423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q" event={"ID":"077d1934-8deb-4b4c-8062-7bbd3a121ea7","Type":"ContainerDied","Data":"d357bfee7c2680bdf31c2790264beb8cc1d97aa9e1b13f203d7fad0c263db71f"}
Sep 16 18:53:28.104479 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.104446    2575 scope.go:117] "RemoveContainer" containerID="89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4"
Sep 16 18:53:28.112950 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.112931    2575 scope.go:117] "RemoveContainer" containerID="89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4"
Sep 16 18:53:28.113222 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:53:28.113206    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4\": container with ID starting with 89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4 not found: ID does not exist" containerID="89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4"
Sep 16 18:53:28.113267 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.113231    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4"} err="failed to get container status \"89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4\": rpc error: code = NotFound desc = could not find container \"89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4\": container with ID starting with 89378865e49e19256dcfc4de538ae17cfb808a5613ff48ba84ee2cc8e2c116b4 not found: ID does not exist"
Sep 16 18:53:28.144815 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.144790    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"]
Sep 16 18:53:28.156043 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.156011    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9kv4q"]
Sep 16 18:53:28.620759 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:28.620703    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="077d1934-8deb-4b4c-8062-7bbd3a121ea7" path="/var/lib/kubelet/pods/077d1934-8deb-4b4c-8062-7bbd3a121ea7/volumes"
Sep 16 18:53:45.657340 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.657306    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tqnq5"]
Sep 16 18:53:45.657741 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.657666    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="077d1934-8deb-4b4c-8062-7bbd3a121ea7" containerName="manager"
Sep 16 18:53:45.657741 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.657678    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="077d1934-8deb-4b4c-8062-7bbd3a121ea7" containerName="manager"
Sep 16 18:53:45.657819 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.657793    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="077d1934-8deb-4b4c-8062-7bbd3a121ea7" containerName="manager"
Sep 16 18:53:45.663428 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.663410    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:45.666247 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.666226    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rc6fs\""
Sep 16 18:53:45.666498 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.666479    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:53:45.677282 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.677260    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tqnq5"]
Sep 16 18:53:45.752301 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.752263    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tqnq5"]
Sep 16 18:53:45.777522 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.777489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73d99494-068b-4b57-8d31-a77a1b20af69-config-file\") pod \"limitador-limitador-8588749977-tqnq5\" (UID: \"73d99494-068b-4b57-8d31-a77a1b20af69\") " pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:45.777522 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.777522    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfkql\" (UniqueName: \"kubernetes.io/projected/73d99494-068b-4b57-8d31-a77a1b20af69-kube-api-access-dfkql\") pod \"limitador-limitador-8588749977-tqnq5\" (UID: \"73d99494-068b-4b57-8d31-a77a1b20af69\") " pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:45.878458 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.878415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73d99494-068b-4b57-8d31-a77a1b20af69-config-file\") pod \"limitador-limitador-8588749977-tqnq5\" (UID: \"73d99494-068b-4b57-8d31-a77a1b20af69\") " pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:45.878458 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.878455    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfkql\" (UniqueName: \"kubernetes.io/projected/73d99494-068b-4b57-8d31-a77a1b20af69-kube-api-access-dfkql\") pod \"limitador-limitador-8588749977-tqnq5\" (UID: \"73d99494-068b-4b57-8d31-a77a1b20af69\") " pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:45.879075 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.879055    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73d99494-068b-4b57-8d31-a77a1b20af69-config-file\") pod \"limitador-limitador-8588749977-tqnq5\" (UID: \"73d99494-068b-4b57-8d31-a77a1b20af69\") " pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:45.888166 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.888136    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfkql\" (UniqueName: \"kubernetes.io/projected/73d99494-068b-4b57-8d31-a77a1b20af69-kube-api-access-dfkql\") pod \"limitador-limitador-8588749977-tqnq5\" (UID: \"73d99494-068b-4b57-8d31-a77a1b20af69\") " pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:45.973170 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:45.973084    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:46.144902 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.144861    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:53:46.149395 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.149371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:46.161594 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.161572    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:53:46.201264 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.201233    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:53:46.247364 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.247269    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tqnq5"]
Sep 16 18:53:46.250198 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:53:46.250175    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d99494_068b_4b57_8d31_a77a1b20af69.slice/crio-ea485714f19a8104afdeab23d400c7481e569cd1db98047aca889fcfe7e1c664 WatchSource:0}: Error finding container ea485714f19a8104afdeab23d400c7481e569cd1db98047aca889fcfe7e1c664: Status 404 returned error can't find the container with id ea485714f19a8104afdeab23d400c7481e569cd1db98047aca889fcfe7e1c664
Sep 16 18:53:46.283896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.283860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgb8v\" (UniqueName: \"kubernetes.io/projected/41aca1be-34b3-426b-b830-ccf47457fde8-kube-api-access-qgb8v\") pod \"limitador-limitador-84994f7ffc-mjhvr\" (UID: \"41aca1be-34b3-426b-b830-ccf47457fde8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:46.284051 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.283907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41aca1be-34b3-426b-b830-ccf47457fde8-config-file\") pod \"limitador-limitador-84994f7ffc-mjhvr\" (UID: \"41aca1be-34b3-426b-b830-ccf47457fde8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:46.385160 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.385119    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41aca1be-34b3-426b-b830-ccf47457fde8-config-file\") pod \"limitador-limitador-84994f7ffc-mjhvr\" (UID: \"41aca1be-34b3-426b-b830-ccf47457fde8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:46.385331 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.385259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgb8v\" (UniqueName: \"kubernetes.io/projected/41aca1be-34b3-426b-b830-ccf47457fde8-kube-api-access-qgb8v\") pod \"limitador-limitador-84994f7ffc-mjhvr\" (UID: \"41aca1be-34b3-426b-b830-ccf47457fde8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:46.385902 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.385880    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41aca1be-34b3-426b-b830-ccf47457fde8-config-file\") pod \"limitador-limitador-84994f7ffc-mjhvr\" (UID: \"41aca1be-34b3-426b-b830-ccf47457fde8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:46.393365 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.393332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgb8v\" (UniqueName: \"kubernetes.io/projected/41aca1be-34b3-426b-b830-ccf47457fde8-kube-api-access-qgb8v\") pod \"limitador-limitador-84994f7ffc-mjhvr\" (UID: \"41aca1be-34b3-426b-b830-ccf47457fde8\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:46.460205 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.460169    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:46.734733 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:46.734628    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:53:46.737407 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:53:46.737382    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41aca1be_34b3_426b_b830_ccf47457fde8.slice/crio-8ce5db7368143f5495802771df3bc8fa5e88fafab96b8e690035280977f57e8c WatchSource:0}: Error finding container 8ce5db7368143f5495802771df3bc8fa5e88fafab96b8e690035280977f57e8c: Status 404 returned error can't find the container with id 8ce5db7368143f5495802771df3bc8fa5e88fafab96b8e690035280977f57e8c
Sep 16 18:53:47.192397 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:47.192345    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5" event={"ID":"73d99494-068b-4b57-8d31-a77a1b20af69","Type":"ContainerStarted","Data":"ea485714f19a8104afdeab23d400c7481e569cd1db98047aca889fcfe7e1c664"}
Sep 16 18:53:47.193820 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:47.193787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr" event={"ID":"41aca1be-34b3-426b-b830-ccf47457fde8","Type":"ContainerStarted","Data":"8ce5db7368143f5495802771df3bc8fa5e88fafab96b8e690035280977f57e8c"}
Sep 16 18:53:48.199447 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:48.199415    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5" event={"ID":"73d99494-068b-4b57-8d31-a77a1b20af69","Type":"ContainerStarted","Data":"a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537"}
Sep 16 18:53:48.199881 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:48.199502    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:48.201093 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:48.201066    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr" event={"ID":"41aca1be-34b3-426b-b830-ccf47457fde8","Type":"ContainerStarted","Data":"72423bb337d47ee6012ea2907975d09a61d8a6ad271fd93fcf45867348cd658c"}
Sep 16 18:53:48.201186 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:48.201169    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:48.231612 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:48.231575    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5" podStartSLOduration=2.006664216 podStartE2EDuration="3.23156117s" totalImagesPullingTime="1.224896954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:48.230329944 +0000 UTC m=+856.047095531" watchObservedRunningTime="2026-09-16 18:53:48.23156117 +0000 UTC m=+856.048326755"
Sep 16 18:53:48.267609 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:48.267563    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr" podStartSLOduration=1.529149246 podStartE2EDuration="2.267550272s" totalImagesPullingTime="738.401026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:48.262612179 +0000 UTC m=+856.079377776" watchObservedRunningTime="2026-09-16 18:53:48.267550272 +0000 UTC m=+856.084315858"
Sep 16 18:53:59.206335 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:59.206302    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:53:59.206848 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:59.206389    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"
Sep 16 18:53:59.268065 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:59.268027    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tqnq5"]
Sep 16 18:53:59.268259 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:53:59.268233    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5" podUID="73d99494-068b-4b57-8d31-a77a1b20af69" containerName="limitador" containerID="cri-o://a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537" gracePeriod=30
Sep 16 18:54:00.209881 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.209854    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:54:00.264388 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.264354    2575 generic.go:356] "Generic (PLEG): container finished" podID="73d99494-068b-4b57-8d31-a77a1b20af69" containerID="a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537" exitCode=0
Sep 16 18:54:00.264541 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.264412    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5"
Sep 16 18:54:00.264541 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.264421    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5" event={"ID":"73d99494-068b-4b57-8d31-a77a1b20af69","Type":"ContainerDied","Data":"a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537"}
Sep 16 18:54:00.264541 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.264500    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tqnq5" event={"ID":"73d99494-068b-4b57-8d31-a77a1b20af69","Type":"ContainerDied","Data":"ea485714f19a8104afdeab23d400c7481e569cd1db98047aca889fcfe7e1c664"}
Sep 16 18:54:00.264541 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.264523    2575 scope.go:117] "RemoveContainer" containerID="a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537"
Sep 16 18:54:00.273144 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.272964    2575 scope.go:117] "RemoveContainer" containerID="a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537"
Sep 16 18:54:00.273288 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:54:00.273269    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537\": container with ID starting with a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537 not found: ID does not exist" containerID="a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537"
Sep 16 18:54:00.273333 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.273297    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537"} err="failed to get container status \"a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537\": rpc error: code = NotFound desc = could not find container \"a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537\": container with ID starting with a6ea6e3a04e2424d99ed5faefc00f0273fdc983cd3c656d241f33f878b1ab537 not found: ID does not exist"
Sep 16 18:54:00.306675 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.306641    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73d99494-068b-4b57-8d31-a77a1b20af69-config-file\") pod \"73d99494-068b-4b57-8d31-a77a1b20af69\" (UID: \"73d99494-068b-4b57-8d31-a77a1b20af69\") "
Sep 16 18:54:00.306849 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.306688    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfkql\" (UniqueName: \"kubernetes.io/projected/73d99494-068b-4b57-8d31-a77a1b20af69-kube-api-access-dfkql\") pod \"73d99494-068b-4b57-8d31-a77a1b20af69\" (UID: \"73d99494-068b-4b57-8d31-a77a1b20af69\") "
Sep 16 18:54:00.307025 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.307004    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d99494-068b-4b57-8d31-a77a1b20af69-config-file" (OuterVolumeSpecName: "config-file") pod "73d99494-068b-4b57-8d31-a77a1b20af69" (UID: "73d99494-068b-4b57-8d31-a77a1b20af69"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:00.308916 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.308893    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d99494-068b-4b57-8d31-a77a1b20af69-kube-api-access-dfkql" (OuterVolumeSpecName: "kube-api-access-dfkql") pod "73d99494-068b-4b57-8d31-a77a1b20af69" (UID: "73d99494-068b-4b57-8d31-a77a1b20af69"). InnerVolumeSpecName "kube-api-access-dfkql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:00.407866 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.407817    2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73d99494-068b-4b57-8d31-a77a1b20af69-config-file\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:54:00.407866 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.407860    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfkql\" (UniqueName: \"kubernetes.io/projected/73d99494-068b-4b57-8d31-a77a1b20af69-kube-api-access-dfkql\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:54:00.589526 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.589487    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tqnq5"]
Sep 16 18:54:00.593359 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.593331    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tqnq5"]
Sep 16 18:54:00.620962 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:00.620930    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73d99494-068b-4b57-8d31-a77a1b20af69" path="/var/lib/kubelet/pods/73d99494-068b-4b57-8d31-a77a1b20af69/volumes"
Sep 16 18:54:23.412927 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.412896    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77f68c45c8-tb6lw"]
Sep 16 18:54:23.413300 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.413287    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73d99494-068b-4b57-8d31-a77a1b20af69" containerName="limitador"
Sep 16 18:54:23.413342 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.413301    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="73d99494-068b-4b57-8d31-a77a1b20af69" containerName="limitador"
Sep 16 18:54:23.413380 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.413357    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="73d99494-068b-4b57-8d31-a77a1b20af69" containerName="limitador"
Sep 16 18:54:23.418442 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.418416    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:23.420948 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.420925    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 18:54:23.421092 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.420925    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zgghk\""
Sep 16 18:54:23.421092 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.420994    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 18:54:23.421786 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.421767    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 18:54:23.426110 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.426090    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f68c45c8-tb6lw"]
Sep 16 18:54:23.506961 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.506937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzlwp\" (UniqueName: \"kubernetes.io/projected/e3593095-0fcf-47c1-96a0-826bb797e787-kube-api-access-qzlwp\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:23.507088 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.507065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-metrics-certs\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:23.507143 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.507114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-webhook-cert\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:23.608505 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.608473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzlwp\" (UniqueName: \"kubernetes.io/projected/e3593095-0fcf-47c1-96a0-826bb797e787-kube-api-access-qzlwp\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:23.608668 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.608540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-metrics-certs\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:23.608668 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.608569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-webhook-cert\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:23.608817 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:54:23.608792    2575 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 16 18:54:23.608892 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:54:23.608880    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-metrics-certs podName:e3593095-0fcf-47c1-96a0-826bb797e787 nodeName:}" failed. No retries permitted until 2026-09-16 18:54:24.108858439 +0000 UTC m=+891.925624018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-metrics-certs") pod "maas-controller-77f68c45c8-tb6lw" (UID: "e3593095-0fcf-47c1-96a0-826bb797e787") : secret "maas-controller-metrics-tls" not found
Sep 16 18:54:23.611207 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.611182    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-webhook-cert\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:23.617373 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.617351    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66cf5649fd-md4ps"]
Sep 16 18:54:23.621815 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.621789    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.622116 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.622097    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzlwp\" (UniqueName: \"kubernetes.io/projected/e3593095-0fcf-47c1-96a0-826bb797e787-kube-api-access-qzlwp\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:23.632960 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.632927    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-md4ps"]
Sep 16 18:54:23.710096 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.710022    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/45b0b69f-cde8-485a-921c-41f0678f3477-kube-api-access-cwh2n\") pod \"maas-controller-66cf5649fd-md4ps\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") " pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.710229 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.710153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-metrics-certs\") pod \"maas-controller-66cf5649fd-md4ps\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") " pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.710229 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.710222    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-webhook-cert\") pod \"maas-controller-66cf5649fd-md4ps\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") " pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.811368 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.811327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-webhook-cert\") pod \"maas-controller-66cf5649fd-md4ps\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") " pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.811510 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.811381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/45b0b69f-cde8-485a-921c-41f0678f3477-kube-api-access-cwh2n\") pod \"maas-controller-66cf5649fd-md4ps\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") " pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.811510 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.811492    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-metrics-certs\") pod \"maas-controller-66cf5649fd-md4ps\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") " pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.813811 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.813785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-metrics-certs\") pod \"maas-controller-66cf5649fd-md4ps\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") " pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.813906 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.813835    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-webhook-cert\") pod \"maas-controller-66cf5649fd-md4ps\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") " pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.819263 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.819245    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/45b0b69f-cde8-485a-921c-41f0678f3477-kube-api-access-cwh2n\") pod \"maas-controller-66cf5649fd-md4ps\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") " pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:23.939832 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:23.939806    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:24.114119 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:24.114085    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-metrics-certs\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:24.116397 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:24.116371    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-metrics-certs\") pod \"maas-controller-77f68c45c8-tb6lw\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") " pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:24.219969 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:24.219935    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-md4ps"]
Sep 16 18:54:24.221418 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:54:24.221388    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-bad492b858de5833e11d4f93a29a4c8316aa6f7b8baf7ba83e6a15cdf6199118 WatchSource:0}: Error finding container bad492b858de5833e11d4f93a29a4c8316aa6f7b8baf7ba83e6a15cdf6199118: Status 404 returned error can't find the container with id bad492b858de5833e11d4f93a29a4c8316aa6f7b8baf7ba83e6a15cdf6199118
Sep 16 18:54:24.352334 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:24.352300    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:24.387984 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:24.387949    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-md4ps" event={"ID":"45b0b69f-cde8-485a-921c-41f0678f3477","Type":"ContainerStarted","Data":"bad492b858de5833e11d4f93a29a4c8316aa6f7b8baf7ba83e6a15cdf6199118"}
Sep 16 18:54:24.625579 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:24.625556    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77f68c45c8-tb6lw"]
Sep 16 18:54:24.627696 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:54:24.627670    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3593095_0fcf_47c1_96a0_826bb797e787.slice/crio-0c866f055c51ffbb38047b9b37bd49d0e31d54ce1ca57a3fb90895f5ca8c374e WatchSource:0}: Error finding container 0c866f055c51ffbb38047b9b37bd49d0e31d54ce1ca57a3fb90895f5ca8c374e: Status 404 returned error can't find the container with id 0c866f055c51ffbb38047b9b37bd49d0e31d54ce1ca57a3fb90895f5ca8c374e
Sep 16 18:54:25.398641 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:25.398600    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f68c45c8-tb6lw" event={"ID":"e3593095-0fcf-47c1-96a0-826bb797e787","Type":"ContainerStarted","Data":"0c866f055c51ffbb38047b9b37bd49d0e31d54ce1ca57a3fb90895f5ca8c374e"}
Sep 16 18:54:27.410445 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:27.410408    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-md4ps" event={"ID":"45b0b69f-cde8-485a-921c-41f0678f3477","Type":"ContainerStarted","Data":"c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121"}
Sep 16 18:54:27.410934 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:27.410544    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:27.412205 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:27.412176    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f68c45c8-tb6lw" event={"ID":"e3593095-0fcf-47c1-96a0-826bb797e787","Type":"ContainerStarted","Data":"bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6"}
Sep 16 18:54:27.412328 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:27.412230    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:27.440178 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:27.440136    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66cf5649fd-md4ps" podStartSLOduration=1.749055912 podStartE2EDuration="4.440123704s" totalImagesPullingTime="2.691067792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:27.436650331 +0000 UTC m=+895.253415915" watchObservedRunningTime="2026-09-16 18:54:27.440123704 +0000 UTC m=+895.256889289"
Sep 16 18:54:27.457274 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:27.457234    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77f68c45c8-tb6lw" podStartSLOduration=2.166105366 podStartE2EDuration="4.45722073s" totalImagesPullingTime="2.291115364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:27.456145406 +0000 UTC m=+895.272910991" watchObservedRunningTime="2026-09-16 18:54:27.45722073 +0000 UTC m=+895.273986315"
Sep 16 18:54:38.419533 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.419498    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:38.419947 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.419586    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 18:54:38.487314 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.487280    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77f68c45c8-tb6lw"]
Sep 16 18:54:38.487516 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.487485    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77f68c45c8-tb6lw" podUID="e3593095-0fcf-47c1-96a0-826bb797e787" containerName="manager" containerID="cri-o://bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6" gracePeriod=30
Sep 16 18:54:38.731397 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.731373    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:38.851271 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.851245    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-metrics-certs\") pod \"e3593095-0fcf-47c1-96a0-826bb797e787\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") "
Sep 16 18:54:38.851398 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.851281    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzlwp\" (UniqueName: \"kubernetes.io/projected/e3593095-0fcf-47c1-96a0-826bb797e787-kube-api-access-qzlwp\") pod \"e3593095-0fcf-47c1-96a0-826bb797e787\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") "
Sep 16 18:54:38.851398 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.851357    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-webhook-cert\") pod \"e3593095-0fcf-47c1-96a0-826bb797e787\" (UID: \"e3593095-0fcf-47c1-96a0-826bb797e787\") "
Sep 16 18:54:38.853216 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.853191    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3593095-0fcf-47c1-96a0-826bb797e787" (UID: "e3593095-0fcf-47c1-96a0-826bb797e787"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:38.853336 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.853266    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3593095-0fcf-47c1-96a0-826bb797e787-kube-api-access-qzlwp" (OuterVolumeSpecName: "kube-api-access-qzlwp") pod "e3593095-0fcf-47c1-96a0-826bb797e787" (UID: "e3593095-0fcf-47c1-96a0-826bb797e787"). InnerVolumeSpecName "kube-api-access-qzlwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:38.853385 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.853356    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e3593095-0fcf-47c1-96a0-826bb797e787" (UID: "e3593095-0fcf-47c1-96a0-826bb797e787"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:38.952366 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.952342    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:54:38.952366 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.952365    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzlwp\" (UniqueName: \"kubernetes.io/projected/e3593095-0fcf-47c1-96a0-826bb797e787-kube-api-access-qzlwp\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:54:38.952484 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:38.952373    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3593095-0fcf-47c1-96a0-826bb797e787-webhook-cert\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:54:39.476366 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:39.476333    2575 generic.go:356] "Generic (PLEG): container finished" podID="e3593095-0fcf-47c1-96a0-826bb797e787" containerID="bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6" exitCode=0
Sep 16 18:54:39.476778 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:39.476374    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77f68c45c8-tb6lw"
Sep 16 18:54:39.476778 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:39.476411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f68c45c8-tb6lw" event={"ID":"e3593095-0fcf-47c1-96a0-826bb797e787","Type":"ContainerDied","Data":"bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6"}
Sep 16 18:54:39.476778 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:39.476463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77f68c45c8-tb6lw" event={"ID":"e3593095-0fcf-47c1-96a0-826bb797e787","Type":"ContainerDied","Data":"0c866f055c51ffbb38047b9b37bd49d0e31d54ce1ca57a3fb90895f5ca8c374e"}
Sep 16 18:54:39.476778 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:39.476486    2575 scope.go:117] "RemoveContainer" containerID="bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6"
Sep 16 18:54:39.485801 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:39.485786    2575 scope.go:117] "RemoveContainer" containerID="bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6"
Sep 16 18:54:39.486057 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:54:39.486040    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6\": container with ID starting with bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6 not found: ID does not exist" containerID="bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6"
Sep 16 18:54:39.486112 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:39.486066    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6"} err="failed to get container status \"bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6\": rpc error: code = NotFound desc = could not find container \"bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6\": container with ID starting with bb0fbeb7882f6fa1fb6e818fe91b74eda77aaacacdc6705f0d5b315b76b30ca6 not found: ID does not exist"
Sep 16 18:54:39.499295 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:39.499272    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77f68c45c8-tb6lw"]
Sep 16 18:54:39.508434 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:39.508403    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77f68c45c8-tb6lw"]
Sep 16 18:54:40.620554 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:54:40.620523    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3593095-0fcf-47c1-96a0-826bb797e787" path="/var/lib/kubelet/pods/e3593095-0fcf-47c1-96a0-826bb797e787/volumes"
Sep 16 18:55:00.471642 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.471607    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"]
Sep 16 18:55:00.472061 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.472047    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3593095-0fcf-47c1-96a0-826bb797e787" containerName="manager"
Sep 16 18:55:00.472105 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.472062    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3593095-0fcf-47c1-96a0-826bb797e787" containerName="manager"
Sep 16 18:55:00.472144 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.472117    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3593095-0fcf-47c1-96a0-826bb797e787" containerName="manager"
Sep 16 18:55:00.479446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.479426    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.484372 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.484330    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:55:00.484913 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.484890    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 18:55:00.485082 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.484924    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 18:55:00.485178 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.484924    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 18:55:00.485260 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.484932    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6cp49\""
Sep 16 18:55:00.487272 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.487124    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"]
Sep 16 18:55:00.539444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.539408    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-maas-api-tls\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.539444 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.539449    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vbt\" (UniqueName: \"kubernetes.io/projected/2aab65ef-c31d-47bf-9123-238b210481bd-kube-api-access-b8vbt\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.539670 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.539470    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aab65ef-c31d-47bf-9123-238b210481bd-openshift-service-ca\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.539670 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.539610    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-metrics-certs\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.640199 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.640160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-maas-api-tls\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.640199 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.640202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vbt\" (UniqueName: \"kubernetes.io/projected/2aab65ef-c31d-47bf-9123-238b210481bd-kube-api-access-b8vbt\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.640456 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.640223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aab65ef-c31d-47bf-9123-238b210481bd-openshift-service-ca\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.640456 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.640416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-metrics-certs\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.641042 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.641022    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aab65ef-c31d-47bf-9123-238b210481bd-openshift-service-ca\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.642720 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.642685    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-metrics-certs\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.643132 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.643116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-maas-api-tls\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.648535 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.648505    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vbt\" (UniqueName: \"kubernetes.io/projected/2aab65ef-c31d-47bf-9123-238b210481bd-kube-api-access-b8vbt\") pod \"maas-api-65b6d4d5dc-p8697\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") " pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:00.791229 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:00.791137    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:01.026030 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:01.025995    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:55:01.088834 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:01.084428    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"]
Sep 16 18:55:01.091956 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:55:01.091914    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aab65ef_c31d_47bf_9123_238b210481bd.slice/crio-83b2ec7ee1409ffb26c81f91f6305a126284f14b79827ad5d60b99262c3fe08a WatchSource:0}: Error finding container 83b2ec7ee1409ffb26c81f91f6305a126284f14b79827ad5d60b99262c3fe08a: Status 404 returned error can't find the container with id 83b2ec7ee1409ffb26c81f91f6305a126284f14b79827ad5d60b99262c3fe08a
Sep 16 18:55:01.586624 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:01.586589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697" event={"ID":"2aab65ef-c31d-47bf-9123-238b210481bd","Type":"ContainerStarted","Data":"83b2ec7ee1409ffb26c81f91f6305a126284f14b79827ad5d60b99262c3fe08a"}
Sep 16 18:55:03.599515 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:03.599472    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697" event={"ID":"2aab65ef-c31d-47bf-9123-238b210481bd","Type":"ContainerStarted","Data":"67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b"}
Sep 16 18:55:03.599911 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:03.599603    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:03.620262 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:03.620206    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697" podStartSLOduration=1.90011643 podStartE2EDuration="3.620189021s" totalImagesPullingTime="1.720072591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:03.61793207 +0000 UTC m=+931.434697678" watchObservedRunningTime="2026-09-16 18:55:03.620189021 +0000 UTC m=+931.436954606"
Sep 16 18:55:09.609758 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:09.609703    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 18:55:18.876086 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:18.876052    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"]
Sep 16 18:55:18.879691 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:18.879674    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:18.882282 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:18.882258    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 18:55:18.883076 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:18.883053    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 18:55:18.883177 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:18.883078    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 18:55:18.883177 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:18.883078    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lhtq7\""
Sep 16 18:55:18.892585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:18.892562    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"]
Sep 16 18:55:19.007882 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.007845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.008045 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.007949    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.008045 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.007984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.008145 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.008061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.008145 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.008095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5p2x\" (UniqueName: \"kubernetes.io/projected/0c57185a-5650-4090-80c3-70763c111d18-kube-api-access-v5p2x\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.008145 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.008122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c57185a-5650-4090-80c3-70763c111d18-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.107922 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.107880    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:55:19.108535 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.108510    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.108652 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.108544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.108652 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.108588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.108652 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.108609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5p2x\" (UniqueName: \"kubernetes.io/projected/0c57185a-5650-4090-80c3-70763c111d18-kube-api-access-v5p2x\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.108652 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.108631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c57185a-5650-4090-80c3-70763c111d18-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.108907 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.108661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.109062 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.109039    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.109130 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.109075    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.109260 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.109234    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.110824 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.110799    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c57185a-5650-4090-80c3-70763c111d18-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.111175 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.111157    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c57185a-5650-4090-80c3-70763c111d18-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.119497 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.119477    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5p2x\" (UniqueName: \"kubernetes.io/projected/0c57185a-5650-4090-80c3-70763c111d18-kube-api-access-v5p2x\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85\" (UID: \"0c57185a-5650-4090-80c3-70763c111d18\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.191752 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.191649    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:19.465390 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.465364    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"]
Sep 16 18:55:19.468093 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:55:19.468065    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c57185a_5650_4090_80c3_70763c111d18.slice/crio-0ec63a4a6d66d3bb84f2a96d68b900515343501a397d2e83dc36ab12bac7fbf2 WatchSource:0}: Error finding container 0ec63a4a6d66d3bb84f2a96d68b900515343501a397d2e83dc36ab12bac7fbf2: Status 404 returned error can't find the container with id 0ec63a4a6d66d3bb84f2a96d68b900515343501a397d2e83dc36ab12bac7fbf2
Sep 16 18:55:19.684136 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:19.684094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85" event={"ID":"0c57185a-5650-4090-80c3-70763c111d18","Type":"ContainerStarted","Data":"0ec63a4a6d66d3bb84f2a96d68b900515343501a397d2e83dc36ab12bac7fbf2"}
Sep 16 18:55:20.690395 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:20.690351    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85" event={"ID":"0c57185a-5650-4090-80c3-70763c111d18","Type":"ContainerStarted","Data":"d02b3a042e5b283954d88a87c9b6601e4b33f1f93ae405ec62a618e524670ecd"}
Sep 16 18:55:20.710033 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:20.709977    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85" podStartSLOduration=1.693376518 podStartE2EDuration="2.709957574s" totalImagesPullingTime="1.016581056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:20.709417857 +0000 UTC m=+948.526183468" watchObservedRunningTime="2026-09-16 18:55:20.709957574 +0000 UTC m=+948.526723158"
Sep 16 18:55:29.192378 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:29.192329    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:29.194746 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:29.192401    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:29.205284 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:29.205256    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:29.745370 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:29.745334    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85"
Sep 16 18:55:34.294786 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:34.294749    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:55:40.568433 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.568398    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"]
Sep 16 18:55:40.574389 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.574363    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.577539 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.577514    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 18:55:40.583941 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.583913    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"]
Sep 16 18:55:40.715154 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.715103    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.715154 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.715164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beeabfbb-beb9-4318-a210-8c9fb293b600-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.715378 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.715291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.715378 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.715312    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.715378 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.715335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcc6\" (UniqueName: \"kubernetes.io/projected/beeabfbb-beb9-4318-a210-8c9fb293b600-kube-api-access-vfcc6\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.715541 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.715513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.816219 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.816177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.816219 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.816218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.816457 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.816242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcc6\" (UniqueName: \"kubernetes.io/projected/beeabfbb-beb9-4318-a210-8c9fb293b600-kube-api-access-vfcc6\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.816457 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.816366    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.816457 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.816446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.816612 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.816469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beeabfbb-beb9-4318-a210-8c9fb293b600-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.816667 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.816645    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.816747 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.816729    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.816881 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.816867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.818666 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.818613    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beeabfbb-beb9-4318-a210-8c9fb293b600-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.818941 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.818924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beeabfbb-beb9-4318-a210-8c9fb293b600-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.825363 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.825337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcc6\" (UniqueName: \"kubernetes.io/projected/beeabfbb-beb9-4318-a210-8c9fb293b600-kube-api-access-vfcc6\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j\" (UID: \"beeabfbb-beb9-4318-a210-8c9fb293b600\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:40.885893 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:40.885854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:41.171931 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:41.171899    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"]
Sep 16 18:55:41.174664 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:55:41.174634    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeeabfbb_beb9_4318_a210_8c9fb293b600.slice/crio-26177aa59f01901788713ba5c5b9f62188126454b0e98643b4c90f681b697f44 WatchSource:0}: Error finding container 26177aa59f01901788713ba5c5b9f62188126454b0e98643b4c90f681b697f44: Status 404 returned error can't find the container with id 26177aa59f01901788713ba5c5b9f62188126454b0e98643b4c90f681b697f44
Sep 16 18:55:41.506429 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:41.506335    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:55:41.790596 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:41.790503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j" event={"ID":"beeabfbb-beb9-4318-a210-8c9fb293b600","Type":"ContainerStarted","Data":"03677fbe4cf27b30d48d4d0e4c335f50a575f836db4848038174b180cadda994"}
Sep 16 18:55:41.790596 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:41.790560    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j" event={"ID":"beeabfbb-beb9-4318-a210-8c9fb293b600","Type":"ContainerStarted","Data":"26177aa59f01901788713ba5c5b9f62188126454b0e98643b4c90f681b697f44"}
Sep 16 18:55:41.812673 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:41.812617    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j" podStartSLOduration=1.63529218 podStartE2EDuration="1.812597296s" totalImagesPullingTime="177.305116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:41.80985857 +0000 UTC m=+969.626624158" watchObservedRunningTime="2026-09-16 18:55:41.812597296 +0000 UTC m=+969.629362882"
Sep 16 18:55:50.886874 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:50.886822    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:50.887271 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:50.886909    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:50.899562 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:50.899532    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:55:51.852323 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:55:51.852292    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j"
Sep 16 18:56:05.406050 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:56:05.405966    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:56:11.705822 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:56:11.705786    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:56:18.517231 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:56:18.517179    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:56:33.998895 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:56:33.998855    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:58:14.318341 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.318301    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7235-d1d8f1"]
Sep 16 18:58:14.321913 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.321884    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:14.326323 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.326301    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q2jrz\""
Sep 16 18:58:14.374376 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.374336    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7235-d1d8f1"]
Sep 16 18:58:14.396111 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.396082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgr5\" (UniqueName: \"kubernetes.io/projected/76083f02-9d7c-42d9-a223-f36fd40f18d7-kube-api-access-8zgr5\") pod \"test-curl-7235-d1d8f1\" (UID: \"76083f02-9d7c-42d9-a223-f36fd40f18d7\") " pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:14.496813 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.496773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgr5\" (UniqueName: \"kubernetes.io/projected/76083f02-9d7c-42d9-a223-f36fd40f18d7-kube-api-access-8zgr5\") pod \"test-curl-7235-d1d8f1\" (UID: \"76083f02-9d7c-42d9-a223-f36fd40f18d7\") " pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:14.513060 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.513029    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgr5\" (UniqueName: \"kubernetes.io/projected/76083f02-9d7c-42d9-a223-f36fd40f18d7-kube-api-access-8zgr5\") pod \"test-curl-7235-d1d8f1\" (UID: \"76083f02-9d7c-42d9-a223-f36fd40f18d7\") " pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:14.631417 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.631384    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:14.942843 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.942818    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7235-d1d8f1"]
Sep 16 18:58:14.945619 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:14.945592    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76083f02_9d7c_42d9_a223_f36fd40f18d7.slice/crio-85ca8b8c03182b56890ac3c2a9b327bcc12afdf5a1417e9fb6075d00ee925c2a WatchSource:0}: Error finding container 85ca8b8c03182b56890ac3c2a9b327bcc12afdf5a1417e9fb6075d00ee925c2a: Status 404 returned error can't find the container with id 85ca8b8c03182b56890ac3c2a9b327bcc12afdf5a1417e9fb6075d00ee925c2a
Sep 16 18:58:14.947466 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:14.947449    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:58:15.556245 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:15.556206    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7235-d1d8f1" event={"ID":"76083f02-9d7c-42d9-a223-f36fd40f18d7","Type":"ContainerStarted","Data":"85ca8b8c03182b56890ac3c2a9b327bcc12afdf5a1417e9fb6075d00ee925c2a"}
Sep 16 18:58:16.341948 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:16.341909    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:58:18.070935 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.070840    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"]
Sep 16 18:58:18.074520 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.074498    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.077145 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.077124    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b1857c-serving-cert\""
Sep 16 18:58:18.090735 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.090696    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"]
Sep 16 18:58:18.132622 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.132595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9sv6\" (UniqueName: \"kubernetes.io/projected/f34a3c69-14d6-4d29-bc3a-711647a01b5a-kube-api-access-h9sv6\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.132737 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.132630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-metrics-certs\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.132737 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.132725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.132863 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.132764    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34a3c69-14d6-4d29-bc3a-711647a01b5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.233949 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.233916    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9sv6\" (UniqueName: \"kubernetes.io/projected/f34a3c69-14d6-4d29-bc3a-711647a01b5a-kube-api-access-h9sv6\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.234098 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.233957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-metrics-certs\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.234098 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.234021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.234098 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.234045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34a3c69-14d6-4d29-bc3a-711647a01b5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.234208 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:18.234147    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-serving-cert: secret "maas-api-e2e-worker-w0-b1857c-serving-cert" not found
Sep 16 18:58:18.234243 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:18.234211    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-maas-api-tls podName:f34a3c69-14d6-4d29-bc3a-711647a01b5a nodeName:}" failed. No retries permitted until 2026-09-16 18:58:18.73419335 +0000 UTC m=+1126.550958918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-maas-api-tls") pod "maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7" (UID: "f34a3c69-14d6-4d29-bc3a-711647a01b5a") : secret "maas-api-e2e-worker-w0-b1857c-serving-cert" not found
Sep 16 18:58:18.234690 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.234670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34a3c69-14d6-4d29-bc3a-711647a01b5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.236680 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.236657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-metrics-certs\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.245797 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.245776    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9sv6\" (UniqueName: \"kubernetes.io/projected/f34a3c69-14d6-4d29-bc3a-711647a01b5a-kube-api-access-h9sv6\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.574183 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.574148    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7235-d1d8f1" event={"ID":"76083f02-9d7c-42d9-a223-f36fd40f18d7","Type":"ContainerStarted","Data":"9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544"}
Sep 16 18:58:18.591875 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.591829    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7235-d1d8f1" podStartSLOduration=1.914008127 podStartE2EDuration="4.591812806s" totalImagesPullingTime="2.677804679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:18.59093425 +0000 UTC m=+1126.407699838" watchObservedRunningTime="2026-09-16 18:58:18.591812806 +0000 UTC m=+1126.408578392"
Sep 16 18:58:18.739597 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.739566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.741942 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.741918    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:18.984621 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:18.984594    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:19.266206 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.266181    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"]
Sep 16 18:58:19.267949 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:19.267919    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34a3c69_14d6_4d29_bc3a_711647a01b5a.slice/crio-ec8c1c2592c4ddad229947ace6d6db62b726ae7db60835ac42453cc39de81885 WatchSource:0}: Error finding container ec8c1c2592c4ddad229947ace6d6db62b726ae7db60835ac42453cc39de81885: Status 404 returned error can't find the container with id ec8c1c2592c4ddad229947ace6d6db62b726ae7db60835ac42453cc39de81885
Sep 16 18:58:19.315483 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.315452    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7235-d1d8f1"]
Sep 16 18:58:19.318785 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.318761    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7235-d1d8f1"]
Sep 16 18:58:19.440982 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.440929    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7235-95df89"]
Sep 16 18:58:19.441374 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.441359    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" containerName="test-curl-7235-d1d8f1"
Sep 16 18:58:19.441435 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.441376    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" containerName="test-curl-7235-d1d8f1"
Sep 16 18:58:19.441498 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.441439    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" containerName="test-curl-7235-d1d8f1"
Sep 16 18:58:19.445613 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.445591    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:19.455073 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.455053    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7235-95df89"]
Sep 16 18:58:19.495419 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.495303    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"]
Sep 16 18:58:19.499031 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.499009    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.501418 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.501396    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ec32e7-serving-cert\""
Sep 16 18:58:19.515911 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.515843    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"]
Sep 16 18:58:19.547113 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.547087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77s28\" (UniqueName: \"kubernetes.io/projected/4045611c-804b-4f72-988d-f264c8bc3153-kube-api-access-77s28\") pod \"test-curl-7235-95df89\" (UID: \"4045611c-804b-4f72-988d-f264c8bc3153\") " pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:19.547244 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.547153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qkf\" (UniqueName: \"kubernetes.io/projected/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-kube-api-access-89qkf\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.547244 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.547205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.547244 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.547230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-metrics-certs\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.547357 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.547299    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.586418 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.586380    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7" event={"ID":"f34a3c69-14d6-4d29-bc3a-711647a01b5a","Type":"ContainerStarted","Data":"ec8c1c2592c4ddad229947ace6d6db62b726ae7db60835ac42453cc39de81885"}
Sep 16 18:58:19.586593 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:19.586575    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7235-d1d8f1" container="test-curl-7235-d1d8f1"
Sep 16 18:58:19.648413 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.648383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77s28\" (UniqueName: \"kubernetes.io/projected/4045611c-804b-4f72-988d-f264c8bc3153-kube-api-access-77s28\") pod \"test-curl-7235-95df89\" (UID: \"4045611c-804b-4f72-988d-f264c8bc3153\") " pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:19.648559 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.648440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qkf\" (UniqueName: \"kubernetes.io/projected/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-kube-api-access-89qkf\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.648634 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.648559    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.648634 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.648623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-metrics-certs\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.648772 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.648680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.648823 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:19.648785    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-serving-cert: secret "maas-api-e2e-inf-a-ec32e7-serving-cert" not found
Sep 16 18:58:19.648866 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:19.648856    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-maas-api-tls podName:d6afb5c7-ce99-4f4a-a296-ac1cf372d7af nodeName:}" failed. No retries permitted until 2026-09-16 18:58:20.148836069 +0000 UTC m=+1127.965601638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-maas-api-tls") pod "maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42" (UID: "d6afb5c7-ce99-4f4a-a296-ac1cf372d7af") : secret "maas-api-e2e-inf-a-ec32e7-serving-cert" not found
Sep 16 18:58:19.649261 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.649241    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.651030 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.651007    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-metrics-certs\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.661359 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.661339    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77s28\" (UniqueName: \"kubernetes.io/projected/4045611c-804b-4f72-988d-f264c8bc3153-kube-api-access-77s28\") pod \"test-curl-7235-95df89\" (UID: \"4045611c-804b-4f72-988d-f264c8bc3153\") " pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:19.661500 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.661483    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qkf\" (UniqueName: \"kubernetes.io/projected/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-kube-api-access-89qkf\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:19.756804 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:19.756777    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:20.033392 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.033355    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7235-95df89"]
Sep 16 18:58:20.036008 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:20.035976    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4045611c_804b_4f72_988d_f264c8bc3153.slice/crio-12a1ef2fa5d6889c8a726512593d941e2b8498de683cf7eca4f58e7a9fafebed WatchSource:0}: Error finding container 12a1ef2fa5d6889c8a726512593d941e2b8498de683cf7eca4f58e7a9fafebed: Status 404 returned error can't find the container with id 12a1ef2fa5d6889c8a726512593d941e2b8498de683cf7eca4f58e7a9fafebed
Sep 16 18:58:20.153009 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.152970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:20.155322 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.155294    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:20.410952 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.410913    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:20.592929 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.592895    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7235-95df89" event={"ID":"4045611c-804b-4f72-988d-f264c8bc3153","Type":"ContainerStarted","Data":"01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec"}
Sep 16 18:58:20.593125 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.592942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7235-95df89" event={"ID":"4045611c-804b-4f72-988d-f264c8bc3153","Type":"ContainerStarted","Data":"12a1ef2fa5d6889c8a726512593d941e2b8498de683cf7eca4f58e7a9fafebed"}
Sep 16 18:58:20.594520 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.594492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7" event={"ID":"f34a3c69-14d6-4d29-bc3a-711647a01b5a","Type":"ContainerStarted","Data":"6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde"}
Sep 16 18:58:20.594632 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.594555    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:20.594762 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.594738    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7235-d1d8f1" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" containerName="test-curl-7235-d1d8f1" containerID="cri-o://9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544" gracePeriod=2
Sep 16 18:58:20.608528 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.608476    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7235-95df89" podStartSLOduration=1.6084581199999999 podStartE2EDuration="1.60845812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:20.607661656 +0000 UTC m=+1128.424427243" watchObservedRunningTime="2026-09-16 18:58:20.60845812 +0000 UTC m=+1128.425223706"
Sep 16 18:58:20.612833 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:20.612798    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-d1d8f1\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:20.631596 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.631529    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7" podStartSLOduration=2.052945827 podStartE2EDuration="2.631509663s" totalImagesPullingTime="578.563836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:20.629811568 +0000 UTC m=+1128.446577155" watchObservedRunningTime="2026-09-16 18:58:20.631509663 +0000 UTC m=+1128.448275250"
Sep 16 18:58:20.692253 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:20.692226    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"]
Sep 16 18:58:20.693469 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:20.693446    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6afb5c7_ce99_4f4a_a296_ac1cf372d7af.slice/crio-ebe281848f08d3f6ed7688084a8c0beb04b493fb85782a9fa9edb3b3f4faf1ba WatchSource:0}: Error finding container ebe281848f08d3f6ed7688084a8c0beb04b493fb85782a9fa9edb3b3f4faf1ba: Status 404 returned error can't find the container with id ebe281848f08d3f6ed7688084a8c0beb04b493fb85782a9fa9edb3b3f4faf1ba
Sep 16 18:58:21.248294 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.248254    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7235-95df89"]
Sep 16 18:58:21.255500 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.255468    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7235-95df89"]
Sep 16 18:58:21.602317 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.602281    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42" event={"ID":"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af","Type":"ContainerStarted","Data":"00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa"}
Sep 16 18:58:21.602774 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.602336    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42" event={"ID":"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af","Type":"ContainerStarted","Data":"ebe281848f08d3f6ed7688084a8c0beb04b493fb85782a9fa9edb3b3f4faf1ba"}
Sep 16 18:58:21.602774 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.602365    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:21.602774 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:21.602609    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7235-95df89" container="test-curl-7235-95df89"
Sep 16 18:58:21.637856 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.637798    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42" podStartSLOduration=2.262789584 podStartE2EDuration="2.637778101s" totalImagesPullingTime="374.988517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:21.637224495 +0000 UTC m=+1129.453990081" watchObservedRunningTime="2026-09-16 18:58:21.637778101 +0000 UTC m=+1129.454543687"
Sep 16 18:58:21.722792 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.722751    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"]
Sep 16 18:58:21.723271 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.723252    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4045611c-804b-4f72-988d-f264c8bc3153" containerName="test-curl-7235-95df89"
Sep 16 18:58:21.723355 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.723273    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4045611c-804b-4f72-988d-f264c8bc3153" containerName="test-curl-7235-95df89"
Sep 16 18:58:21.723418 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.723375    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4045611c-804b-4f72-988d-f264c8bc3153" containerName="test-curl-7235-95df89"
Sep 16 18:58:21.726646 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.726624    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.729682 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.729657    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c090c8-serving-cert\""
Sep 16 18:58:21.738757 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.738735    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"]
Sep 16 18:58:21.869644 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.869543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-metrics-certs\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.869644 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.869610    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8a36e-c095-4321-9878-04b47e7b7072-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.869912 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.869698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrk5z\" (UniqueName: \"kubernetes.io/projected/18b8a36e-c095-4321-9878-04b47e7b7072-kube-api-access-hrk5z\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.869912 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.869900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.970784 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.970750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.970980 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.970799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-metrics-certs\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.970980 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.970838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8a36e-c095-4321-9878-04b47e7b7072-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.970980 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.970891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrk5z\" (UniqueName: \"kubernetes.io/projected/18b8a36e-c095-4321-9878-04b47e7b7072-kube-api-access-hrk5z\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.970980 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:21.970904    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-serving-cert: secret "maas-api-e2e-worker-w4-c090c8-serving-cert" not found
Sep 16 18:58:21.970980 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:21.970969    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-maas-api-tls podName:18b8a36e-c095-4321-9878-04b47e7b7072 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:22.470948731 +0000 UTC m=+1130.287714307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-maas-api-tls") pod "maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb" (UID: "18b8a36e-c095-4321-9878-04b47e7b7072") : secret "maas-api-e2e-worker-w4-c090c8-serving-cert" not found
Sep 16 18:58:21.971668 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.971636    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8a36e-c095-4321-9878-04b47e7b7072-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.973841 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.973819    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-metrics-certs\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:21.986042 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:21.985997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrk5z\" (UniqueName: \"kubernetes.io/projected/18b8a36e-c095-4321-9878-04b47e7b7072-kube-api-access-hrk5z\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:22.475413 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.475376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:22.478861 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.478836    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:22.608698 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.608655    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7235-95df89" podUID="4045611c-804b-4f72-988d-f264c8bc3153" containerName="test-curl-7235-95df89" containerID="cri-o://01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec" gracePeriod=2
Sep 16 18:58:22.610988 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:22.610957    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-95df89\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="4045611c-804b-4f72-988d-f264c8bc3153" pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:22.619116 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:22.619091    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-95df89\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="4045611c-804b-4f72-988d-f264c8bc3153" pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:22.636955 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.636937    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:22.825314 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.825287    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:22.827244 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:22.827210    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-d1d8f1\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:22.878588 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.878554    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zgr5\" (UniqueName: \"kubernetes.io/projected/76083f02-9d7c-42d9-a223-f36fd40f18d7-kube-api-access-8zgr5\") pod \"76083f02-9d7c-42d9-a223-f36fd40f18d7\" (UID: \"76083f02-9d7c-42d9-a223-f36fd40f18d7\") "
Sep 16 18:58:22.882575 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.882544    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76083f02-9d7c-42d9-a223-f36fd40f18d7-kube-api-access-8zgr5" (OuterVolumeSpecName: "kube-api-access-8zgr5") pod "76083f02-9d7c-42d9-a223-f36fd40f18d7" (UID: "76083f02-9d7c-42d9-a223-f36fd40f18d7"). InnerVolumeSpecName "kube-api-access-8zgr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:22.935324 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.935301    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"]
Sep 16 18:58:22.938531 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:22.938498    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b8a36e_c095_4321_9878_04b47e7b7072.slice/crio-45542a474bdca088ab08729c74107110f2d731b506ad2eceb647245973f59767 WatchSource:0}: Error finding container 45542a474bdca088ab08729c74107110f2d731b506ad2eceb647245973f59767: Status 404 returned error can't find the container with id 45542a474bdca088ab08729c74107110f2d731b506ad2eceb647245973f59767
Sep 16 18:58:22.959704 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.959673    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"]
Sep 16 18:58:22.963566 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.963543    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:22.968039 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.968007    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c090c8\""
Sep 16 18:58:22.968581 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.968561    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c090c8-dockercfg-v6lbq\""
Sep 16 18:58:22.976368 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.976345    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"]
Sep 16 18:58:22.979890 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:22.979865    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zgr5\" (UniqueName: \"kubernetes.io/projected/76083f02-9d7c-42d9-a223-f36fd40f18d7-kube-api-access-8zgr5\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:58:22.995943 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:22.995911    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-d1d8f1\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:23.080911 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.080832    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c67554-1901-4306-b10c-99602e43c9bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw\" (UID: \"24c67554-1901-4306-b10c-99602e43c9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:23.080911 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.080870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnpcp\" (UniqueName: \"kubernetes.io/projected/24c67554-1901-4306-b10c-99602e43c9bc-kube-api-access-xnpcp\") pod \"payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw\" (UID: \"24c67554-1901-4306-b10c-99602e43c9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:23.182232 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.182192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c67554-1901-4306-b10c-99602e43c9bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw\" (UID: \"24c67554-1901-4306-b10c-99602e43c9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:23.182232 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.182231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnpcp\" (UniqueName: \"kubernetes.io/projected/24c67554-1901-4306-b10c-99602e43c9bc-kube-api-access-xnpcp\") pod \"payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw\" (UID: \"24c67554-1901-4306-b10c-99602e43c9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:23.182923 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.182903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c67554-1901-4306-b10c-99602e43c9bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw\" (UID: \"24c67554-1901-4306-b10c-99602e43c9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:23.191445 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.191425    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnpcp\" (UniqueName: \"kubernetes.io/projected/24c67554-1901-4306-b10c-99602e43c9bc-kube-api-access-xnpcp\") pod \"payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw\" (UID: \"24c67554-1901-4306-b10c-99602e43c9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:23.275674 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.275654    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:23.615188 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.615160    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb" event={"ID":"18b8a36e-c095-4321-9878-04b47e7b7072","Type":"ContainerStarted","Data":"a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d"}
Sep 16 18:58:23.615614 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.615350    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:23.615614 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.615388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb" event={"ID":"18b8a36e-c095-4321-9878-04b47e7b7072","Type":"ContainerStarted","Data":"45542a474bdca088ab08729c74107110f2d731b506ad2eceb647245973f59767"}
Sep 16 18:58:23.616419 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.616390    2575 generic.go:356] "Generic (PLEG): container finished" podID="76083f02-9d7c-42d9-a223-f36fd40f18d7" containerID="9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544" exitCode=137
Sep 16 18:58:23.616538 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.616443    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:23.616538 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.616457    2575 scope.go:117] "RemoveContainer" containerID="9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544"
Sep 16 18:58:23.625728 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.625691    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"]
Sep 16 18:58:23.626820 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.626799    2575 scope.go:117] "RemoveContainer" containerID="9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544"
Sep 16 18:58:23.627080 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:23.627060    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544\": container with ID starting with 9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544 not found: ID does not exist" containerID="9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544"
Sep 16 18:58:23.627162 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.627085    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544"} err="failed to get container status \"9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544\": rpc error: code = NotFound desc = could not find container \"9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544\": container with ID starting with 9817c445a9bf77034409b9de67124c11bbcdd462b7631e556c6132359c84b544 not found: ID does not exist"
Sep 16 18:58:23.628207 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:23.628186    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c67554_1901_4306_b10c_99602e43c9bc.slice/crio-7b9614cd6003abf03a68b410cb0ac8df30ba77a48c494412976317d17c424e32 WatchSource:0}: Error finding container 7b9614cd6003abf03a68b410cb0ac8df30ba77a48c494412976317d17c424e32: Status 404 returned error can't find the container with id 7b9614cd6003abf03a68b410cb0ac8df30ba77a48c494412976317d17c424e32
Sep 16 18:58:23.682748 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:23.682670    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb" podStartSLOduration=2.371737947 podStartE2EDuration="2.682649603s" totalImagesPullingTime="310.911656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:23.679281164 +0000 UTC m=+1131.496046761" watchObservedRunningTime="2026-09-16 18:58:23.682649603 +0000 UTC m=+1131.499415191"
Sep 16 18:58:23.684664 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:23.684634    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-d1d8f1\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:23.686800 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:23.686770    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-d1d8f1\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" pod="opendatahub/test-curl-7235-d1d8f1"
Sep 16 18:58:24.626881 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:24.626746    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76083f02-9d7c-42d9-a223-f36fd40f18d7" path="/var/lib/kubelet/pods/76083f02-9d7c-42d9-a223-f36fd40f18d7/volumes"
Sep 16 18:58:24.627655 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:24.627625    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw" event={"ID":"24c67554-1901-4306-b10c-99602e43c9bc","Type":"ContainerStarted","Data":"7b9614cd6003abf03a68b410cb0ac8df30ba77a48c494412976317d17c424e32"}
Sep 16 18:58:25.083931 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.083908    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:25.201963 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.201937    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77s28\" (UniqueName: \"kubernetes.io/projected/4045611c-804b-4f72-988d-f264c8bc3153-kube-api-access-77s28\") pod \"4045611c-804b-4f72-988d-f264c8bc3153\" (UID: \"4045611c-804b-4f72-988d-f264c8bc3153\") "
Sep 16 18:58:25.203871 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.203852    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4045611c-804b-4f72-988d-f264c8bc3153-kube-api-access-77s28" (OuterVolumeSpecName: "kube-api-access-77s28") pod "4045611c-804b-4f72-988d-f264c8bc3153" (UID: "4045611c-804b-4f72-988d-f264c8bc3153"). InnerVolumeSpecName "kube-api-access-77s28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:25.303570 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.303537    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77s28\" (UniqueName: \"kubernetes.io/projected/4045611c-804b-4f72-988d-f264c8bc3153-kube-api-access-77s28\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:58:25.635685 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.635646    2575 generic.go:356] "Generic (PLEG): container finished" podID="4045611c-804b-4f72-988d-f264c8bc3153" containerID="01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec" exitCode=137
Sep 16 18:58:25.636159 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.635693    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:25.636159 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.635759    2575 scope.go:117] "RemoveContainer" containerID="01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec"
Sep 16 18:58:25.637786 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.637760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw" event={"ID":"24c67554-1901-4306-b10c-99602e43c9bc","Type":"ContainerStarted","Data":"63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200"}
Sep 16 18:58:25.637926 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.637875    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:25.645841 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.645818    2575 scope.go:117] "RemoveContainer" containerID="01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec"
Sep 16 18:58:25.646168 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:25.646149    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec\": container with ID starting with 01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec not found: ID does not exist" containerID="01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec"
Sep 16 18:58:25.646226 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.646178    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec"} err="failed to get container status \"01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec\": rpc error: code = NotFound desc = could not find container \"01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec\": container with ID starting with 01e06082eebcc0fdaaa1d10b6599f8b4c05a3d5c384a0f1ff75b2b65d06f2fec not found: ID does not exist"
Sep 16 18:58:25.661953 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:25.661914    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-95df89\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="4045611c-804b-4f72-988d-f264c8bc3153" pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:25.662106 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:25.661955    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw" podStartSLOduration=2.183919982 podStartE2EDuration="3.661935264s" totalImagesPullingTime="1.478015282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:25.659857113 +0000 UTC m=+1133.476622700" watchObservedRunningTime="2026-09-16 18:58:25.661935264 +0000 UTC m=+1133.478700852"
Sep 16 18:58:26.608363 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:26.608333    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 18:58:26.622696 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:26.622662    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4045611c-804b-4f72-988d-f264c8bc3153" path="/var/lib/kubelet/pods/4045611c-804b-4f72-988d-f264c8bc3153/volumes"
Sep 16 18:58:26.629999 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:26.629965    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-95df89\" is forbidden: User \"system:node:ip-10-0-138-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-234.ec2.internal' and this object" podUID="4045611c-804b-4f72-988d-f264c8bc3153" pod="opendatahub/test-curl-7235-95df89"
Sep 16 18:58:27.160856 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.160817    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"]
Sep 16 18:58:27.166443 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.166415    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.168972 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.168946    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-2efa6f-serving-cert\""
Sep 16 18:58:27.178943 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.178914    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"]
Sep 16 18:58:27.326329 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.326293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftc2t\" (UniqueName: \"kubernetes.io/projected/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-kube-api-access-ftc2t\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.326500 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.326340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-metrics-certs\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.326500 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.326439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-openshift-service-ca\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.326500 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.326469    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-maas-api-tls\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.427563 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.427490    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-openshift-service-ca\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.427563 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.427532    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-maas-api-tls\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.427755 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.427590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftc2t\" (UniqueName: \"kubernetes.io/projected/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-kube-api-access-ftc2t\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.427755 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.427613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-metrics-certs\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.427755 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:27.427733    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-serving-cert: secret "maas-api-e2e-autores-2efa6f-serving-cert" not found
Sep 16 18:58:27.427871 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:27.427806    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-maas-api-tls podName:d20a83d1-f8a4-4ef7-ae41-930ef65e3669 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:27.927784477 +0000 UTC m=+1135.744550047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-maas-api-tls") pod "maas-api-e2e-autores-2efa6f-648597db5f-j27lr" (UID: "d20a83d1-f8a4-4ef7-ae41-930ef65e3669") : secret "maas-api-e2e-autores-2efa6f-serving-cert" not found
Sep 16 18:58:27.428183 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.428164    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-openshift-service-ca\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.430087 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.430059    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-metrics-certs\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.446494 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.446473    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftc2t\" (UniqueName: \"kubernetes.io/projected/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-kube-api-access-ftc2t\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.614241 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.614210    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 18:58:27.934184 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.934150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-maas-api-tls\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:27.936738 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:27.936690    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-maas-api-tls\") pod \"maas-api-e2e-autores-2efa6f-648597db5f-j27lr\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:28.080099 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:28.080060    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:28.368993 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:28.368966    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"]
Sep 16 18:58:28.370788 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:28.370761    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20a83d1_f8a4_4ef7_ae41_930ef65e3669.slice/crio-a50130ff2057fc1be3b26422167367984fef62ec9a012886685559ee2d66ccaa WatchSource:0}: Error finding container a50130ff2057fc1be3b26422167367984fef62ec9a012886685559ee2d66ccaa: Status 404 returned error can't find the container with id a50130ff2057fc1be3b26422167367984fef62ec9a012886685559ee2d66ccaa
Sep 16 18:58:28.660847 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:28.660757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr" event={"ID":"d20a83d1-f8a4-4ef7-ae41-930ef65e3669","Type":"ContainerStarted","Data":"a50130ff2057fc1be3b26422167367984fef62ec9a012886685559ee2d66ccaa"}
Sep 16 18:58:29.632978 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:29.632944    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 18:58:29.670077 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:29.670038    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr" event={"ID":"d20a83d1-f8a4-4ef7-ae41-930ef65e3669","Type":"ContainerStarted","Data":"c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c"}
Sep 16 18:58:29.670261 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:29.670107    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:29.705340 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:29.705281    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr" podStartSLOduration=2.314195808 podStartE2EDuration="2.705258119s" totalImagesPullingTime="391.062311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:29.703114614 +0000 UTC m=+1137.519880201" watchObservedRunningTime="2026-09-16 18:58:29.705258119 +0000 UTC m=+1137.522023706"
Sep 16 18:58:30.566609 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.566572    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"]
Sep 16 18:58:30.572299 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.572267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.574954 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.574746    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4300c3c2-serving-cert\""
Sep 16 18:58:30.581887 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.581859    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"]
Sep 16 18:58:30.661748 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.661693    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xwmf\" (UniqueName: \"kubernetes.io/projected/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-kube-api-access-5xwmf\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.662206 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.661844    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-metrics-certs\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.662206 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.661917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-maas-api-tls\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.662206 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.661985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-openshift-service-ca\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.763497 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.763456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xwmf\" (UniqueName: \"kubernetes.io/projected/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-kube-api-access-5xwmf\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.763694 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.763553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-metrics-certs\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.763694 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.763623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-maas-api-tls\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.763694 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.763651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-openshift-service-ca\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.763956 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:30.763933    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-serving-cert: secret "maas-api-e2e-ait-4300c3c2-serving-cert" not found
Sep 16 18:58:30.764038 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:30.764003    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-maas-api-tls podName:a9bd142a-eaeb-4913-8ada-05fdf4c572ec nodeName:}" failed. No retries permitted until 2026-09-16 18:58:31.263982592 +0000 UTC m=+1139.080748155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-maas-api-tls") pod "maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz" (UID: "a9bd142a-eaeb-4913-8ada-05fdf4c572ec") : secret "maas-api-e2e-ait-4300c3c2-serving-cert" not found
Sep 16 18:58:30.764547 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.764527    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-openshift-service-ca\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.766108 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.766087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-metrics-certs\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:30.775268 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:30.775240    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xwmf\" (UniqueName: \"kubernetes.io/projected/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-kube-api-access-5xwmf\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:31.270070 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.270032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-maas-api-tls\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:31.272511 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.272492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-maas-api-tls\") pod \"maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:31.485828 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.485792    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:31.681427 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.681386    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"]
Sep 16 18:58:31.688106 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.688073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:31.691303 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.691277    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4300c3c2-dockercfg-v9s55\""
Sep 16 18:58:31.691434 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.691277    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4300c3c2\""
Sep 16 18:58:31.707984 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.707952    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"]
Sep 16 18:58:31.774435 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.774395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq8px\" (UniqueName: \"kubernetes.io/projected/f69386e9-e6d9-477b-867e-f91461d9ca36-kube-api-access-nq8px\") pod \"payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4\" (UID: \"f69386e9-e6d9-477b-867e-f91461d9ca36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:31.774586 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.774489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69386e9-e6d9-477b-867e-f91461d9ca36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4\" (UID: \"f69386e9-e6d9-477b-867e-f91461d9ca36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:31.786945 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.786862    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"]
Sep 16 18:58:31.799416 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:31.799390    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bd142a_eaeb_4913_8ada_05fdf4c572ec.slice/crio-16f3096080d8108f87eef55596277f69dbc87a6fc7ec402c9fc6ff18bbf302d0 WatchSource:0}: Error finding container 16f3096080d8108f87eef55596277f69dbc87a6fc7ec402c9fc6ff18bbf302d0: Status 404 returned error can't find the container with id 16f3096080d8108f87eef55596277f69dbc87a6fc7ec402c9fc6ff18bbf302d0
Sep 16 18:58:31.875420 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.875390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8px\" (UniqueName: \"kubernetes.io/projected/f69386e9-e6d9-477b-867e-f91461d9ca36-kube-api-access-nq8px\") pod \"payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4\" (UID: \"f69386e9-e6d9-477b-867e-f91461d9ca36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:31.875597 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.875474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69386e9-e6d9-477b-867e-f91461d9ca36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4\" (UID: \"f69386e9-e6d9-477b-867e-f91461d9ca36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:31.876238 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.876218    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69386e9-e6d9-477b-867e-f91461d9ca36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4\" (UID: \"f69386e9-e6d9-477b-867e-f91461d9ca36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:31.902454 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:31.902421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8px\" (UniqueName: \"kubernetes.io/projected/f69386e9-e6d9-477b-867e-f91461d9ca36-kube-api-access-nq8px\") pod \"payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4\" (UID: \"f69386e9-e6d9-477b-867e-f91461d9ca36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:32.000837 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.000797    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:32.151288 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.151247    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:58:32.289082 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.289053    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"]
Sep 16 18:58:32.292522 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:32.292490    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69386e9_e6d9_477b_867e_f91461d9ca36.slice/crio-b1832db86c45c1282f0a27f045c3ccdde61329d8090b0ff64eb20db920ce1a88 WatchSource:0}: Error finding container b1832db86c45c1282f0a27f045c3ccdde61329d8090b0ff64eb20db920ce1a88: Status 404 returned error can't find the container with id b1832db86c45c1282f0a27f045c3ccdde61329d8090b0ff64eb20db920ce1a88
Sep 16 18:58:32.689591 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.689561    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz" event={"ID":"a9bd142a-eaeb-4913-8ada-05fdf4c572ec","Type":"ContainerStarted","Data":"899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567"}
Sep 16 18:58:32.689956 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.689617    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz" event={"ID":"a9bd142a-eaeb-4913-8ada-05fdf4c572ec","Type":"ContainerStarted","Data":"16f3096080d8108f87eef55596277f69dbc87a6fc7ec402c9fc6ff18bbf302d0"}
Sep 16 18:58:32.689956 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.689645    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:32.691121 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.691098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4" event={"ID":"f69386e9-e6d9-477b-867e-f91461d9ca36","Type":"ContainerStarted","Data":"58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3"}
Sep 16 18:58:32.691215 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.691136    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4" event={"ID":"f69386e9-e6d9-477b-867e-f91461d9ca36","Type":"ContainerStarted","Data":"b1832db86c45c1282f0a27f045c3ccdde61329d8090b0ff64eb20db920ce1a88"}
Sep 16 18:58:32.691215 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.691184    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:32.723486 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:32.723388    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz" podStartSLOduration=2.198575702 podStartE2EDuration="2.72336843s" totalImagesPullingTime="524.792728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:32.723019479 +0000 UTC m=+1140.539785067" watchObservedRunningTime="2026-09-16 18:58:32.72336843 +0000 UTC m=+1140.540134021"
Sep 16 18:58:35.681652 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:35.681621    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 18:58:35.720446 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:35.720390    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4" podStartSLOduration=4.720373133 podStartE2EDuration="4.720373133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:32.75389543 +0000 UTC m=+1140.570661017" watchObservedRunningTime="2026-09-16 18:58:35.720373133 +0000 UTC m=+1143.537138769"
Sep 16 18:58:36.649817 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:36.649786    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 18:58:38.701666 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:38.701639    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:43.698503 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:43.698472    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:45.971654 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:45.971601    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"]
Sep 16 18:58:45.980808 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:45.980780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:45.982992 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:45.982967    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8a45a9-serving-cert\""
Sep 16 18:58:45.989986 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:45.989956    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"]
Sep 16 18:58:46.014342 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.014302    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.014518 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.014356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c1592-509d-47c9-ad1b-607c73d3ea59-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.014518 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.014477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-metrics-certs\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.014634 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.014528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrc2g\" (UniqueName: \"kubernetes.io/projected/ea1c1592-509d-47c9-ad1b-607c73d3ea59-kube-api-access-xrc2g\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.115702 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.115623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrc2g\" (UniqueName: \"kubernetes.io/projected/ea1c1592-509d-47c9-ad1b-607c73d3ea59-kube-api-access-xrc2g\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.115937 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.115740    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.115937 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.115789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c1592-509d-47c9-ad1b-607c73d3ea59-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.115937 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.115875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-metrics-certs\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.115937 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:46.115888    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-serving-cert: secret "maas-api-e2e-inf-b-8a45a9-serving-cert" not found
Sep 16 18:58:46.116172 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:46.115962    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-maas-api-tls podName:ea1c1592-509d-47c9-ad1b-607c73d3ea59 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:46.615941892 +0000 UTC m=+1154.432707461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-maas-api-tls") pod "maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c" (UID: "ea1c1592-509d-47c9-ad1b-607c73d3ea59") : secret "maas-api-e2e-inf-b-8a45a9-serving-cert" not found
Sep 16 18:58:46.116662 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.116639    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c1592-509d-47c9-ad1b-607c73d3ea59-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.118371 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.118352    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-metrics-certs\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.137190 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.137166    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrc2g\" (UniqueName: \"kubernetes.io/projected/ea1c1592-509d-47c9-ad1b-607c73d3ea59-kube-api-access-xrc2g\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.619871 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.619830    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.622890 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.622858    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.877248 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.877150    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"]
Sep 16 18:58:46.881595 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.881569    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:58:46.887619 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.887593    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8a45a9\""
Sep 16 18:58:46.893541 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.893514    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:46.896632 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.896605    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"]
Sep 16 18:58:46.923386 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.923345    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z456g\" (UniqueName: \"kubernetes.io/projected/ea2108f5-f17c-4377-9730-2f990ef6b4b2-kube-api-access-z456g\") pod \"payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98\" (UID: \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:58:46.923625 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.923597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2108f5-f17c-4377-9730-2f990ef6b4b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98\" (UID: \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:58:46.924585 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.924513    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"]
Sep 16 18:58:46.931764 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.931695    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:46.933363 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.933332    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"]
Sep 16 18:58:46.934425 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:46.934396    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8a45a9-dockercfg-gnx84\""
Sep 16 18:58:47.024286 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.024246    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fwqv\" (UniqueName: \"kubernetes.io/projected/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-kube-api-access-8fwqv\") pod \"payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6\" (UID: \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:47.024786 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.024336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2108f5-f17c-4377-9730-2f990ef6b4b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98\" (UID: \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:58:47.024786 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.024402    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z456g\" (UniqueName: \"kubernetes.io/projected/ea2108f5-f17c-4377-9730-2f990ef6b4b2-kube-api-access-z456g\") pod \"payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98\" (UID: \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:58:47.024786 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.024470    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6\" (UID: \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:47.025196 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.025173    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2108f5-f17c-4377-9730-2f990ef6b4b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98\" (UID: \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:58:47.038246 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.038223    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z456g\" (UniqueName: \"kubernetes.io/projected/ea2108f5-f17c-4377-9730-2f990ef6b4b2-kube-api-access-z456g\") pod \"payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98\" (UID: \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:58:47.125625 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.125585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6\" (UID: \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:47.125843 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.125655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwqv\" (UniqueName: \"kubernetes.io/projected/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-kube-api-access-8fwqv\") pod \"payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6\" (UID: \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:47.126476 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.126451    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6\" (UID: \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:47.136501 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.136434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwqv\" (UniqueName: \"kubernetes.io/projected/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-kube-api-access-8fwqv\") pod \"payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6\" (UID: \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:47.194318 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.194276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:58:47.201415 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.201385    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"]
Sep 16 18:58:47.205633 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:47.205605    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1c1592_509d_47c9_ad1b_607c73d3ea59.slice/crio-7a6434495fd6fa1b25a6f966a5c435826d0403f55472218339b2ea6514f7b871 WatchSource:0}: Error finding container 7a6434495fd6fa1b25a6f966a5c435826d0403f55472218339b2ea6514f7b871: Status 404 returned error can't find the container with id 7a6434495fd6fa1b25a6f966a5c435826d0403f55472218339b2ea6514f7b871
Sep 16 18:58:47.246668 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.246628    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:47.479902 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.479876    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"]
Sep 16 18:58:47.481771 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:47.481744    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2108f5_f17c_4377_9730_2f990ef6b4b2.slice/crio-a111e5a392ff235287eb55954c85d6f4c934fa04def40ef4b3fcd142650271fa WatchSource:0}: Error finding container a111e5a392ff235287eb55954c85d6f4c934fa04def40ef4b3fcd142650271fa: Status 404 returned error can't find the container with id a111e5a392ff235287eb55954c85d6f4c934fa04def40ef4b3fcd142650271fa
Sep 16 18:58:47.553285 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.553245    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"]
Sep 16 18:58:47.555882 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:58:47.555856    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11b2aeb_ecde_47b6_999b_e52aaf7725b7.slice/crio-add78d46a3aa4f04ac02d950bf6b22793b4cb91fc07c739e2da98db2154d669e WatchSource:0}: Error finding container add78d46a3aa4f04ac02d950bf6b22793b4cb91fc07c739e2da98db2154d669e: Status 404 returned error can't find the container with id add78d46a3aa4f04ac02d950bf6b22793b4cb91fc07c739e2da98db2154d669e
Sep 16 18:58:47.786203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.786112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6" event={"ID":"f11b2aeb-ecde-47b6-999b-e52aaf7725b7","Type":"ContainerStarted","Data":"fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298"}
Sep 16 18:58:47.786203 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.786178    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6" event={"ID":"f11b2aeb-ecde-47b6-999b-e52aaf7725b7","Type":"ContainerStarted","Data":"add78d46a3aa4f04ac02d950bf6b22793b4cb91fc07c739e2da98db2154d669e"}
Sep 16 18:58:47.786417 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.786245    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:47.787991 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.787965    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c" event={"ID":"ea1c1592-509d-47c9-ad1b-607c73d3ea59","Type":"ContainerStarted","Data":"4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155"}
Sep 16 18:58:47.788129 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.788011    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c" event={"ID":"ea1c1592-509d-47c9-ad1b-607c73d3ea59","Type":"ContainerStarted","Data":"7a6434495fd6fa1b25a6f966a5c435826d0403f55472218339b2ea6514f7b871"}
Sep 16 18:58:47.788129 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.788066    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:47.789745 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.789695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98" event={"ID":"ea2108f5-f17c-4377-9730-2f990ef6b4b2","Type":"ContainerStarted","Data":"fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7"}
Sep 16 18:58:47.789834 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.789768    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98" event={"ID":"ea2108f5-f17c-4377-9730-2f990ef6b4b2","Type":"ContainerStarted","Data":"a111e5a392ff235287eb55954c85d6f4c934fa04def40ef4b3fcd142650271fa"}
Sep 16 18:58:47.789884 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.789849    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:58:47.811809 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.811762    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6" podStartSLOduration=1.8117457510000001 podStartE2EDuration="1.811745751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:47.809875428 +0000 UTC m=+1155.626641015" watchObservedRunningTime="2026-09-16 18:58:47.811745751 +0000 UTC m=+1155.628511342"
Sep 16 18:58:47.836038 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.835985    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98" podStartSLOduration=1.835965558 podStartE2EDuration="1.835965558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:47.832836725 +0000 UTC m=+1155.649602314" watchObservedRunningTime="2026-09-16 18:58:47.835965558 +0000 UTC m=+1155.652731145"
Sep 16 18:58:47.857897 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:47.857834    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c" podStartSLOduration=2.461670681 podStartE2EDuration="2.857813578s" totalImagesPullingTime="396.142897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:47.857484498 +0000 UTC m=+1155.674250086" watchObservedRunningTime="2026-09-16 18:58:47.857813578 +0000 UTC m=+1155.674579165"
Sep 16 18:58:53.125042 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.125000    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"]
Sep 16 18:58:53.125435 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.125265    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz" podUID="a9bd142a-eaeb-4913-8ada-05fdf4c572ec" containerName="maas-api" containerID="cri-o://899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567" gracePeriod=30
Sep 16 18:58:53.206896 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.206862    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"]
Sep 16 18:58:53.207130 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.207097    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4" podUID="f69386e9-e6d9-477b-867e-f91461d9ca36" containerName="payload-pre-processing" containerID="cri-o://58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3" gracePeriod=30
Sep 16 18:58:53.473974 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.473949    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:53.504364 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.504337    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:53.584928 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.584893    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-metrics-certs\") pod \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") "
Sep 16 18:58:53.585099 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.584946    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69386e9-e6d9-477b-867e-f91461d9ca36-plugins-config-volume\") pod \"f69386e9-e6d9-477b-867e-f91461d9ca36\" (UID: \"f69386e9-e6d9-477b-867e-f91461d9ca36\") "
Sep 16 18:58:53.585099 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.585068    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-maas-api-tls\") pod \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") "
Sep 16 18:58:53.585232 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.585110    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq8px\" (UniqueName: \"kubernetes.io/projected/f69386e9-e6d9-477b-867e-f91461d9ca36-kube-api-access-nq8px\") pod \"f69386e9-e6d9-477b-867e-f91461d9ca36\" (UID: \"f69386e9-e6d9-477b-867e-f91461d9ca36\") "
Sep 16 18:58:53.585232 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.585144    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-openshift-service-ca\") pod \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") "
Sep 16 18:58:53.585232 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.585179    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xwmf\" (UniqueName: \"kubernetes.io/projected/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-kube-api-access-5xwmf\") pod \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\" (UID: \"a9bd142a-eaeb-4913-8ada-05fdf4c572ec\") "
Sep 16 18:58:53.585397 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.585335    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69386e9-e6d9-477b-867e-f91461d9ca36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f69386e9-e6d9-477b-867e-f91461d9ca36" (UID: "f69386e9-e6d9-477b-867e-f91461d9ca36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:53.585867 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.585573    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69386e9-e6d9-477b-867e-f91461d9ca36-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:58:53.585867 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.585826    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9bd142a-eaeb-4913-8ada-05fdf4c572ec" (UID: "a9bd142a-eaeb-4913-8ada-05fdf4c572ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:53.587651 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.587614    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a9bd142a-eaeb-4913-8ada-05fdf4c572ec" (UID: "a9bd142a-eaeb-4913-8ada-05fdf4c572ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:53.587878 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.587850    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69386e9-e6d9-477b-867e-f91461d9ca36-kube-api-access-nq8px" (OuterVolumeSpecName: "kube-api-access-nq8px") pod "f69386e9-e6d9-477b-867e-f91461d9ca36" (UID: "f69386e9-e6d9-477b-867e-f91461d9ca36"). InnerVolumeSpecName "kube-api-access-nq8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:53.587995 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.587939    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9bd142a-eaeb-4913-8ada-05fdf4c572ec" (UID: "a9bd142a-eaeb-4913-8ada-05fdf4c572ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:53.587995 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.587977    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-kube-api-access-5xwmf" (OuterVolumeSpecName: "kube-api-access-5xwmf") pod "a9bd142a-eaeb-4913-8ada-05fdf4c572ec" (UID: "a9bd142a-eaeb-4913-8ada-05fdf4c572ec"). InnerVolumeSpecName "kube-api-access-5xwmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:53.686426 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.686331    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:58:53.686426 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.686365    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:58:53.686426 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.686379    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq8px\" (UniqueName: \"kubernetes.io/projected/f69386e9-e6d9-477b-867e-f91461d9ca36-kube-api-access-nq8px\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:58:53.686426 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.686391    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:58:53.686426 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.686403    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xwmf\" (UniqueName: \"kubernetes.io/projected/a9bd142a-eaeb-4913-8ada-05fdf4c572ec-kube-api-access-5xwmf\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:58:53.801665 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.801636    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 18:58:53.824967 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.824930    2575 generic.go:356] "Generic (PLEG): container finished" podID="f69386e9-e6d9-477b-867e-f91461d9ca36" containerID="58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3" exitCode=0
Sep 16 18:58:53.824967 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.824978    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"
Sep 16 18:58:53.825300 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.825018    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4" event={"ID":"f69386e9-e6d9-477b-867e-f91461d9ca36","Type":"ContainerDied","Data":"58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3"}
Sep 16 18:58:53.825300 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.825074    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4" event={"ID":"f69386e9-e6d9-477b-867e-f91461d9ca36","Type":"ContainerDied","Data":"b1832db86c45c1282f0a27f045c3ccdde61329d8090b0ff64eb20db920ce1a88"}
Sep 16 18:58:53.825300 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.825098    2575 scope.go:117] "RemoveContainer" containerID="58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3"
Sep 16 18:58:53.827921 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.827000    2575 generic.go:356] "Generic (PLEG): container finished" podID="a9bd142a-eaeb-4913-8ada-05fdf4c572ec" containerID="899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567" exitCode=0
Sep 16 18:58:53.827921 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.827068    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz" event={"ID":"a9bd142a-eaeb-4913-8ada-05fdf4c572ec","Type":"ContainerDied","Data":"899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567"}
Sep 16 18:58:53.827921 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.827113    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz" event={"ID":"a9bd142a-eaeb-4913-8ada-05fdf4c572ec","Type":"ContainerDied","Data":"16f3096080d8108f87eef55596277f69dbc87a6fc7ec402c9fc6ff18bbf302d0"}
Sep 16 18:58:53.827921 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.827191    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"
Sep 16 18:58:53.836193 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.836174    2575 scope.go:117] "RemoveContainer" containerID="58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3"
Sep 16 18:58:53.836531 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:53.836510    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3\": container with ID starting with 58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3 not found: ID does not exist" containerID="58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3"
Sep 16 18:58:53.836635 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.836537    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3"} err="failed to get container status \"58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3\": rpc error: code = NotFound desc = could not find container \"58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3\": container with ID starting with 58bc0a87f31db232c0792b67832d90b8ba055510effd7070fac0bbfb6ab148f3 not found: ID does not exist"
Sep 16 18:58:53.836635 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.836554    2575 scope.go:117] "RemoveContainer" containerID="899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567"
Sep 16 18:58:53.844792 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.844774    2575 scope.go:117] "RemoveContainer" containerID="899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567"
Sep 16 18:58:53.845086 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:58:53.845063    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567\": container with ID starting with 899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567 not found: ID does not exist" containerID="899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567"
Sep 16 18:58:53.845152 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.845097    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567"} err="failed to get container status \"899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567\": rpc error: code = NotFound desc = could not find container \"899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567\": container with ID starting with 899b80d5692f085bb959b242ff30aed385562b1be827447a1338e00b043ce567 not found: ID does not exist"
Sep 16 18:58:53.865639 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.865611    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"]
Sep 16 18:58:53.871805 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.871776    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4300c3c2-79bb95fd58-t2xn4"]
Sep 16 18:58:53.883351 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.883239    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"]
Sep 16 18:58:53.885421 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:53.885393    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4300c3c2-5978db4896-pz4qz"]
Sep 16 18:58:54.622510 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:54.622475    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9bd142a-eaeb-4913-8ada-05fdf4c572ec" path="/var/lib/kubelet/pods/a9bd142a-eaeb-4913-8ada-05fdf4c572ec/volumes"
Sep 16 18:58:54.622928 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:54.622912    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f69386e9-e6d9-477b-867e-f91461d9ca36" path="/var/lib/kubelet/pods/f69386e9-e6d9-477b-867e-f91461d9ca36/volumes"
Sep 16 18:58:54.942947 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:54.942846    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:58:58.798251 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:58.798218    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 18:58:58.798695 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:58:58.798298    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 18:59:03.032766 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.028563    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"]
Sep 16 18:59:03.032766 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.029785    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f69386e9-e6d9-477b-867e-f91461d9ca36" containerName="payload-pre-processing"
Sep 16 18:59:03.032766 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.029809    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f69386e9-e6d9-477b-867e-f91461d9ca36" containerName="payload-pre-processing"
Sep 16 18:59:03.032766 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.029849    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9bd142a-eaeb-4913-8ada-05fdf4c572ec" containerName="maas-api"
Sep 16 18:59:03.032766 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.029858    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9bd142a-eaeb-4913-8ada-05fdf4c572ec" containerName="maas-api"
Sep 16 18:59:03.032766 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.030061    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9bd142a-eaeb-4913-8ada-05fdf4c572ec" containerName="maas-api"
Sep 16 18:59:03.032766 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.030081    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f69386e9-e6d9-477b-867e-f91461d9ca36" containerName="payload-pre-processing"
Sep 16 18:59:03.035058 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.035029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.041407 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.041383    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-78ad8baa-serving-cert\""
Sep 16 18:59:03.058330 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.058309    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"]
Sep 16 18:59:03.184566 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.184526    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95qq\" (UniqueName: \"kubernetes.io/projected/69467845-7ef9-43bb-b616-6e30e8f18640-kube-api-access-n95qq\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.184566 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.184572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69467845-7ef9-43bb-b616-6e30e8f18640-openshift-service-ca\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.184834 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.184657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-maas-api-tls\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.184834 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.184724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-metrics-certs\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.286054 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.285961    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95qq\" (UniqueName: \"kubernetes.io/projected/69467845-7ef9-43bb-b616-6e30e8f18640-kube-api-access-n95qq\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.286054 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.286002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69467845-7ef9-43bb-b616-6e30e8f18640-openshift-service-ca\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.286054 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.286046    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-maas-api-tls\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.286329 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.286079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-metrics-certs\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.286329 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:59:03.286198    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-serving-cert: secret "maas-api-e2e-migrate-78ad8baa-serving-cert" not found
Sep 16 18:59:03.286329 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:59:03.286281    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-maas-api-tls podName:69467845-7ef9-43bb-b616-6e30e8f18640 nodeName:}" failed. No retries permitted until 2026-09-16 18:59:03.786262614 +0000 UTC m=+1171.603028185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-maas-api-tls") pod "maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2" (UID: "69467845-7ef9-43bb-b616-6e30e8f18640") : secret "maas-api-e2e-migrate-78ad8baa-serving-cert" not found
Sep 16 18:59:03.286687 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.286663    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69467845-7ef9-43bb-b616-6e30e8f18640-openshift-service-ca\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.288611 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.288579    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-metrics-certs\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.300353 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.300323    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95qq\" (UniqueName: \"kubernetes.io/projected/69467845-7ef9-43bb-b616-6e30e8f18640-kube-api-access-n95qq\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.792283 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.792247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-maas-api-tls\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.794911 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.794879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-maas-api-tls\") pod \"maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:03.946315 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:03.946284    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:04.227061 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:04.227033    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"]
Sep 16 18:59:04.228701 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:59:04.228675    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69467845_7ef9_43bb_b616_6e30e8f18640.slice/crio-768fb526e55a4a950b24a70cd718d4649179f6e10dd38c6849cf4e9934a931eb WatchSource:0}: Error finding container 768fb526e55a4a950b24a70cd718d4649179f6e10dd38c6849cf4e9934a931eb: Status 404 returned error can't find the container with id 768fb526e55a4a950b24a70cd718d4649179f6e10dd38c6849cf4e9934a931eb
Sep 16 18:59:04.887843 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:04.887758    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2" event={"ID":"69467845-7ef9-43bb-b616-6e30e8f18640","Type":"ContainerStarted","Data":"d367b4ea27598440c9f12a8bf79709321227a858178a0fd01989914a7804ce2e"}
Sep 16 18:59:04.887843 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:04.887809    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2" event={"ID":"69467845-7ef9-43bb-b616-6e30e8f18640","Type":"ContainerStarted","Data":"768fb526e55a4a950b24a70cd718d4649179f6e10dd38c6849cf4e9934a931eb"}
Sep 16 18:59:04.888057 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:04.887956    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:04.927774 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:04.927725    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2" podStartSLOduration=2.533177248 podStartE2EDuration="2.927695551s" totalImagesPullingTime="394.518303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:04.926810512 +0000 UTC m=+1172.743576097" watchObservedRunningTime="2026-09-16 18:59:04.927695551 +0000 UTC m=+1172.744461137"
Sep 16 18:59:10.899014 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:10.898981    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:13.118878 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:13.118846    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:59:38.489565 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:38.489534    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:59:40.890835 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:40.890797    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"]
Sep 16 18:59:40.891272 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:40.891024    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2" podUID="69467845-7ef9-43bb-b616-6e30e8f18640" containerName="maas-api" containerID="cri-o://d367b4ea27598440c9f12a8bf79709321227a858178a0fd01989914a7804ce2e" gracePeriod=30
Sep 16 18:59:40.958648 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:59:40.958608    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69467845_7ef9_43bb_b616_6e30e8f18640.slice/crio-d367b4ea27598440c9f12a8bf79709321227a858178a0fd01989914a7804ce2e.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 18:59:41.091832 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.091743    2575 generic.go:356] "Generic (PLEG): container finished" podID="69467845-7ef9-43bb-b616-6e30e8f18640" containerID="d367b4ea27598440c9f12a8bf79709321227a858178a0fd01989914a7804ce2e" exitCode=0
Sep 16 18:59:41.091832 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.091817    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2" event={"ID":"69467845-7ef9-43bb-b616-6e30e8f18640","Type":"ContainerDied","Data":"d367b4ea27598440c9f12a8bf79709321227a858178a0fd01989914a7804ce2e"}
Sep 16 18:59:41.178547 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.178517    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:41.279807 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.279764    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69467845-7ef9-43bb-b616-6e30e8f18640-openshift-service-ca\") pod \"69467845-7ef9-43bb-b616-6e30e8f18640\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") "
Sep 16 18:59:41.279807 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.279810    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-maas-api-tls\") pod \"69467845-7ef9-43bb-b616-6e30e8f18640\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") "
Sep 16 18:59:41.280032 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.279876    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n95qq\" (UniqueName: \"kubernetes.io/projected/69467845-7ef9-43bb-b616-6e30e8f18640-kube-api-access-n95qq\") pod \"69467845-7ef9-43bb-b616-6e30e8f18640\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") "
Sep 16 18:59:41.280032 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.279909    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-metrics-certs\") pod \"69467845-7ef9-43bb-b616-6e30e8f18640\" (UID: \"69467845-7ef9-43bb-b616-6e30e8f18640\") "
Sep 16 18:59:41.280242 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.280216    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69467845-7ef9-43bb-b616-6e30e8f18640-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69467845-7ef9-43bb-b616-6e30e8f18640" (UID: "69467845-7ef9-43bb-b616-6e30e8f18640"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:41.282156 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.282123    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "69467845-7ef9-43bb-b616-6e30e8f18640" (UID: "69467845-7ef9-43bb-b616-6e30e8f18640"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:41.282278 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.282172    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69467845-7ef9-43bb-b616-6e30e8f18640-kube-api-access-n95qq" (OuterVolumeSpecName: "kube-api-access-n95qq") pod "69467845-7ef9-43bb-b616-6e30e8f18640" (UID: "69467845-7ef9-43bb-b616-6e30e8f18640"). InnerVolumeSpecName "kube-api-access-n95qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:41.282278 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.282233    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69467845-7ef9-43bb-b616-6e30e8f18640" (UID: "69467845-7ef9-43bb-b616-6e30e8f18640"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:41.381182 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.381139    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n95qq\" (UniqueName: \"kubernetes.io/projected/69467845-7ef9-43bb-b616-6e30e8f18640-kube-api-access-n95qq\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:59:41.381182 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.381174    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:59:41.381182 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.381184    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69467845-7ef9-43bb-b616-6e30e8f18640-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:59:41.381182 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:41.381192    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69467845-7ef9-43bb-b616-6e30e8f18640-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 18:59:42.098539 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:42.098502    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2" event={"ID":"69467845-7ef9-43bb-b616-6e30e8f18640","Type":"ContainerDied","Data":"768fb526e55a4a950b24a70cd718d4649179f6e10dd38c6849cf4e9934a931eb"}
Sep 16 18:59:42.098984 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:42.098560    2575 scope.go:117] "RemoveContainer" containerID="d367b4ea27598440c9f12a8bf79709321227a858178a0fd01989914a7804ce2e"
Sep 16 18:59:42.098984 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:42.098515    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"
Sep 16 18:59:42.131828 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:42.131793    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"]
Sep 16 18:59:42.143121 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:42.143092    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-78ad8baa-5576fc664f-kxsv2"]
Sep 16 18:59:42.621984 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:42.621951    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69467845-7ef9-43bb-b616-6e30e8f18640" path="/var/lib/kubelet/pods/69467845-7ef9-43bb-b616-6e30e8f18640/volumes"
Sep 16 18:59:50.959012 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.958967    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"]
Sep 16 18:59:50.959474 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.959447    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69467845-7ef9-43bb-b616-6e30e8f18640" containerName="maas-api"
Sep 16 18:59:50.959474 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.959459    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="69467845-7ef9-43bb-b616-6e30e8f18640" containerName="maas-api"
Sep 16 18:59:50.959595 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.959554    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="69467845-7ef9-43bb-b616-6e30e8f18640" containerName="maas-api"
Sep 16 18:59:50.964396 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.964370    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:50.968889 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.968866    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8bed97ae-serving-cert\""
Sep 16 18:59:50.975481 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.975458    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-maas-api-tls\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:50.975582 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.975501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4648l\" (UniqueName: \"kubernetes.io/projected/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-kube-api-access-4648l\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:50.975582 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.975526    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-openshift-service-ca\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:50.975700 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.975593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-metrics-certs\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:50.986615 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:50.986587    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"]
Sep 16 18:59:51.076197 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.076161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-maas-api-tls\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:51.076376 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.076208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4648l\" (UniqueName: \"kubernetes.io/projected/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-kube-api-access-4648l\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:51.076376 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:59:51.076309    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-serving-cert: secret "maas-api-e2e-ait-8bed97ae-serving-cert" not found
Sep 16 18:59:51.076376 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.076320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-openshift-service-ca\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:51.076481 ip-10-0-138-234 kubenswrapper[2575]: E0916 18:59:51.076389    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-maas-api-tls podName:d5c1a4d6-edf5-4d9d-a794-a9d671717d97 nodeName:}" failed. No retries permitted until 2026-09-16 18:59:51.576370715 +0000 UTC m=+1219.393136290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-maas-api-tls") pod "maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r" (UID: "d5c1a4d6-edf5-4d9d-a794-a9d671717d97") : secret "maas-api-e2e-ait-8bed97ae-serving-cert" not found
Sep 16 18:59:51.076481 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.076440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-metrics-certs\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:51.077012 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.076992    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-openshift-service-ca\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:51.078842 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.078826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-metrics-certs\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:51.101263 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.101224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4648l\" (UniqueName: \"kubernetes.io/projected/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-kube-api-access-4648l\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:51.581702 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.581659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-maas-api-tls\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:51.584143 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.584122    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-maas-api-tls\") pod \"maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:51.875340 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:51.875306    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:52.169042 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.168964    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"]
Sep 16 18:59:52.173749 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.173729    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 18:59:52.176867 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.176847    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8bed97ae\""
Sep 16 18:59:52.187975 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.187951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ghs\" (UniqueName: \"kubernetes.io/projected/43d40d97-a896-4189-8dfd-23919bddad6e-kube-api-access-x6ghs\") pod \"payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn\" (UID: \"43d40d97-a896-4189-8dfd-23919bddad6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 18:59:52.188060 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.188002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d40d97-a896-4189-8dfd-23919bddad6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn\" (UID: \"43d40d97-a896-4189-8dfd-23919bddad6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 18:59:52.195691 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.195673    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"]
Sep 16 18:59:52.206516 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.206493    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"]
Sep 16 18:59:52.214477 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:59:52.214452    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c1a4d6_edf5_4d9d_a794_a9d671717d97.slice/crio-76f547319692e73b7c4dc4823e44a3301e559811ede9956b8f2006b4625ecb6c WatchSource:0}: Error finding container 76f547319692e73b7c4dc4823e44a3301e559811ede9956b8f2006b4625ecb6c: Status 404 returned error can't find the container with id 76f547319692e73b7c4dc4823e44a3301e559811ede9956b8f2006b4625ecb6c
Sep 16 18:59:52.289475 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.289441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ghs\" (UniqueName: \"kubernetes.io/projected/43d40d97-a896-4189-8dfd-23919bddad6e-kube-api-access-x6ghs\") pod \"payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn\" (UID: \"43d40d97-a896-4189-8dfd-23919bddad6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 18:59:52.289614 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.289526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d40d97-a896-4189-8dfd-23919bddad6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn\" (UID: \"43d40d97-a896-4189-8dfd-23919bddad6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 18:59:52.290342 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.290319    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d40d97-a896-4189-8dfd-23919bddad6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn\" (UID: \"43d40d97-a896-4189-8dfd-23919bddad6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 18:59:52.301769 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.301738    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ghs\" (UniqueName: \"kubernetes.io/projected/43d40d97-a896-4189-8dfd-23919bddad6e-kube-api-access-x6ghs\") pod \"payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn\" (UID: \"43d40d97-a896-4189-8dfd-23919bddad6e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 18:59:52.484392 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.484311    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 18:59:52.785190 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:52.785164    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"]
Sep 16 18:59:52.787402 ip-10-0-138-234 kubenswrapper[2575]: W0916 18:59:52.787371    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d40d97_a896_4189_8dfd_23919bddad6e.slice/crio-c77f2e063193384825994404908fd172f351448038b684d89817eec2e33d4ffe WatchSource:0}: Error finding container c77f2e063193384825994404908fd172f351448038b684d89817eec2e33d4ffe: Status 404 returned error can't find the container with id c77f2e063193384825994404908fd172f351448038b684d89817eec2e33d4ffe
Sep 16 18:59:53.162309 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:53.162269    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r" event={"ID":"d5c1a4d6-edf5-4d9d-a794-a9d671717d97","Type":"ContainerStarted","Data":"9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593"}
Sep 16 18:59:53.162474 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:53.162330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r" event={"ID":"d5c1a4d6-edf5-4d9d-a794-a9d671717d97","Type":"ContainerStarted","Data":"76f547319692e73b7c4dc4823e44a3301e559811ede9956b8f2006b4625ecb6c"}
Sep 16 18:59:53.162474 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:53.162356    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 18:59:53.163812 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:53.163785    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn" event={"ID":"43d40d97-a896-4189-8dfd-23919bddad6e","Type":"ContainerStarted","Data":"1af5a57efe693cd20a7aac24df969030b1f78e76a94513da2cb43bf93859e802"}
Sep 16 18:59:53.163941 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:53.163835    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn" event={"ID":"43d40d97-a896-4189-8dfd-23919bddad6e","Type":"ContainerStarted","Data":"c77f2e063193384825994404908fd172f351448038b684d89817eec2e33d4ffe"}
Sep 16 18:59:53.163941 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:53.163879    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 18:59:53.244935 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:53.244888    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn" podStartSLOduration=1.244870316 podStartE2EDuration="1.244870316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:53.243137589 +0000 UTC m=+1221.059903175" watchObservedRunningTime="2026-09-16 18:59:53.244870316 +0000 UTC m=+1221.061635903"
Sep 16 18:59:53.245483 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:53.245447    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r" podStartSLOduration=2.835325937 podStartE2EDuration="3.245435938s" totalImagesPullingTime="410.110001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:53.201063477 +0000 UTC m=+1221.017829064" watchObservedRunningTime="2026-09-16 18:59:53.245435938 +0000 UTC m=+1221.062201581"
Sep 16 18:59:57.417830 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:57.417791    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 18:59:59.174779 ip-10-0-138-234 kubenswrapper[2575]: I0916 18:59:59.174745    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 19:00:04.171897 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:04.171865    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 19:00:19.438157 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:19.438124    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:00:27.966373 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:27.966281    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"]
Sep 16 19:00:27.966860 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:27.966492    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r" podUID="d5c1a4d6-edf5-4d9d-a794-a9d671717d97" containerName="maas-api" containerID="cri-o://9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593" gracePeriod=30
Sep 16 19:00:28.157825 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.157687    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"]
Sep 16 19:00:28.158013 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.157938    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn" podUID="43d40d97-a896-4189-8dfd-23919bddad6e" containerName="payload-pre-processing" containerID="cri-o://1af5a57efe693cd20a7aac24df969030b1f78e76a94513da2cb43bf93859e802" gracePeriod=30
Sep 16 19:00:28.289420 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.289383    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 19:00:28.357867 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.357824    2575 generic.go:356] "Generic (PLEG): container finished" podID="43d40d97-a896-4189-8dfd-23919bddad6e" containerID="1af5a57efe693cd20a7aac24df969030b1f78e76a94513da2cb43bf93859e802" exitCode=0
Sep 16 19:00:28.358041 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.357905    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn" event={"ID":"43d40d97-a896-4189-8dfd-23919bddad6e","Type":"ContainerDied","Data":"1af5a57efe693cd20a7aac24df969030b1f78e76a94513da2cb43bf93859e802"}
Sep 16 19:00:28.359558 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.359524    2575 generic.go:356] "Generic (PLEG): container finished" podID="d5c1a4d6-edf5-4d9d-a794-a9d671717d97" containerID="9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593" exitCode=0
Sep 16 19:00:28.359724 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.359575    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"
Sep 16 19:00:28.359724 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.359611    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r" event={"ID":"d5c1a4d6-edf5-4d9d-a794-a9d671717d97","Type":"ContainerDied","Data":"9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593"}
Sep 16 19:00:28.359724 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.359660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r" event={"ID":"d5c1a4d6-edf5-4d9d-a794-a9d671717d97","Type":"ContainerDied","Data":"76f547319692e73b7c4dc4823e44a3301e559811ede9956b8f2006b4625ecb6c"}
Sep 16 19:00:28.359724 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.359686    2575 scope.go:117] "RemoveContainer" containerID="9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593"
Sep 16 19:00:28.368339 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.368311    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4648l\" (UniqueName: \"kubernetes.io/projected/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-kube-api-access-4648l\") pod \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") "
Sep 16 19:00:28.368459 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.368375    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-maas-api-tls\") pod \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") "
Sep 16 19:00:28.368459 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.368415    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-metrics-certs\") pod \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") "
Sep 16 19:00:28.368810 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.368786    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-openshift-service-ca\") pod \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\" (UID: \"d5c1a4d6-edf5-4d9d-a794-a9d671717d97\") "
Sep 16 19:00:28.369189 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.369162    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5c1a4d6-edf5-4d9d-a794-a9d671717d97" (UID: "d5c1a4d6-edf5-4d9d-a794-a9d671717d97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:28.369654 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.369630    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.370956 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.370903    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-kube-api-access-4648l" (OuterVolumeSpecName: "kube-api-access-4648l") pod "d5c1a4d6-edf5-4d9d-a794-a9d671717d97" (UID: "d5c1a4d6-edf5-4d9d-a794-a9d671717d97"). InnerVolumeSpecName "kube-api-access-4648l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:28.371049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.370955    2575 scope.go:117] "RemoveContainer" containerID="9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593"
Sep 16 19:00:28.371049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.371018    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5c1a4d6-edf5-4d9d-a794-a9d671717d97" (UID: "d5c1a4d6-edf5-4d9d-a794-a9d671717d97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:28.371268 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.371245    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5c1a4d6-edf5-4d9d-a794-a9d671717d97" (UID: "d5c1a4d6-edf5-4d9d-a794-a9d671717d97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:28.371473 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:00:28.371434    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593\": container with ID starting with 9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593 not found: ID does not exist" containerID="9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593"
Sep 16 19:00:28.371569 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.371484    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593"} err="failed to get container status \"9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593\": rpc error: code = NotFound desc = could not find container \"9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593\": container with ID starting with 9bafbfaa91b48b813de3643976017772dcbc95d779ad7d30f29b3a7f6b907593 not found: ID does not exist"
Sep 16 19:00:28.414418 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.414391    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 19:00:28.470806 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.470772    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6ghs\" (UniqueName: \"kubernetes.io/projected/43d40d97-a896-4189-8dfd-23919bddad6e-kube-api-access-x6ghs\") pod \"43d40d97-a896-4189-8dfd-23919bddad6e\" (UID: \"43d40d97-a896-4189-8dfd-23919bddad6e\") "
Sep 16 19:00:28.470806 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.470809    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d40d97-a896-4189-8dfd-23919bddad6e-plugins-config-volume\") pod \"43d40d97-a896-4189-8dfd-23919bddad6e\" (UID: \"43d40d97-a896-4189-8dfd-23919bddad6e\") "
Sep 16 19:00:28.471040 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.471027    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4648l\" (UniqueName: \"kubernetes.io/projected/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-kube-api-access-4648l\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.471090 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.471042    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.471090 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.471051    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c1a4d6-edf5-4d9d-a794-a9d671717d97-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.471315 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.471286    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d40d97-a896-4189-8dfd-23919bddad6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43d40d97-a896-4189-8dfd-23919bddad6e" (UID: "43d40d97-a896-4189-8dfd-23919bddad6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:28.472857 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.472839    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d40d97-a896-4189-8dfd-23919bddad6e-kube-api-access-x6ghs" (OuterVolumeSpecName: "kube-api-access-x6ghs") pod "43d40d97-a896-4189-8dfd-23919bddad6e" (UID: "43d40d97-a896-4189-8dfd-23919bddad6e"). InnerVolumeSpecName "kube-api-access-x6ghs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:28.571984 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.571886    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ghs\" (UniqueName: \"kubernetes.io/projected/43d40d97-a896-4189-8dfd-23919bddad6e-kube-api-access-x6ghs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.571984 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.571923    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d40d97-a896-4189-8dfd-23919bddad6e-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.693933 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.693900    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"]
Sep 16 19:00:28.700301 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:28.700273    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bed97ae-5b5d668bd5-6mc6r"]
Sep 16 19:00:29.366363 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:29.366327    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"
Sep 16 19:00:29.368627 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:29.366378    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn" event={"ID":"43d40d97-a896-4189-8dfd-23919bddad6e","Type":"ContainerDied","Data":"c77f2e063193384825994404908fd172f351448038b684d89817eec2e33d4ffe"}
Sep 16 19:00:29.368627 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:29.366436    2575 scope.go:117] "RemoveContainer" containerID="1af5a57efe693cd20a7aac24df969030b1f78e76a94513da2cb43bf93859e802"
Sep 16 19:00:29.402926 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:29.402899    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"]
Sep 16 19:00:29.409230 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:29.409197    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bed97ae-756558d9fd-flxnn"]
Sep 16 19:00:30.621776 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:30.621737    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d40d97-a896-4189-8dfd-23919bddad6e" path="/var/lib/kubelet/pods/43d40d97-a896-4189-8dfd-23919bddad6e/volumes"
Sep 16 19:00:30.622192 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:30.622177    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c1a4d6-edf5-4d9d-a794-a9d671717d97" path="/var/lib/kubelet/pods/d5c1a4d6-edf5-4d9d-a794-a9d671717d97/volumes"
Sep 16 19:00:38.888412 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.888377    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"]
Sep 16 19:00:38.888911 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.888869    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d40d97-a896-4189-8dfd-23919bddad6e" containerName="payload-pre-processing"
Sep 16 19:00:38.888911 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.888883    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d40d97-a896-4189-8dfd-23919bddad6e" containerName="payload-pre-processing"
Sep 16 19:00:38.888911 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.888893    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c1a4d6-edf5-4d9d-a794-a9d671717d97" containerName="maas-api"
Sep 16 19:00:38.888911 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.888898    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c1a4d6-edf5-4d9d-a794-a9d671717d97" containerName="maas-api"
Sep 16 19:00:38.889051 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.888970    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c1a4d6-edf5-4d9d-a794-a9d671717d97" containerName="maas-api"
Sep 16 19:00:38.889051 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.888977    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d40d97-a896-4189-8dfd-23919bddad6e" containerName="payload-pre-processing"
Sep 16 19:00:38.893309 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.893291    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:38.897967 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.897947    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-96b42948-serving-cert\""
Sep 16 19:00:38.919181 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.919150    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"]
Sep 16 19:00:38.978691 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.978656    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c896c97a-e86f-4f7b-9384-013e0da2a732-openshift-service-ca\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:38.978900 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.978733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-maas-api-tls\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:38.978900 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.978788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbb9k\" (UniqueName: \"kubernetes.io/projected/c896c97a-e86f-4f7b-9384-013e0da2a732-kube-api-access-bbb9k\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:38.978900 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:38.978815    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-metrics-certs\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.080059 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.080024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c896c97a-e86f-4f7b-9384-013e0da2a732-openshift-service-ca\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.080224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.080071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-maas-api-tls\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.080224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.080114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbb9k\" (UniqueName: \"kubernetes.io/projected/c896c97a-e86f-4f7b-9384-013e0da2a732-kube-api-access-bbb9k\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.080224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.080134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-metrics-certs\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.080835 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.080813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c896c97a-e86f-4f7b-9384-013e0da2a732-openshift-service-ca\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.082566 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.082544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-maas-api-tls\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.082663 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.082584    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-metrics-certs\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.105279 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.105255    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbb9k\" (UniqueName: \"kubernetes.io/projected/c896c97a-e86f-4f7b-9384-013e0da2a732-kube-api-access-bbb9k\") pod \"maas-api-e2e-derive-96b42948-5976996d77-qbz8k\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.203375 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.203278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:39.528534 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:39.528458    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"]
Sep 16 19:00:39.531758 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:00:39.531729    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc896c97a_e86f_4f7b_9384_013e0da2a732.slice/crio-0a97677cc5eb8b4aede565e4b5522690c14d8ccbf18a43b8eaf2e7b05a5c506d WatchSource:0}: Error finding container 0a97677cc5eb8b4aede565e4b5522690c14d8ccbf18a43b8eaf2e7b05a5c506d: Status 404 returned error can't find the container with id 0a97677cc5eb8b4aede565e4b5522690c14d8ccbf18a43b8eaf2e7b05a5c506d
Sep 16 19:00:40.428233 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.428133    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k" event={"ID":"c896c97a-e86f-4f7b-9384-013e0da2a732","Type":"ContainerStarted","Data":"243f5cc0a6eb1549d92ef1a459747502473fa6b9655ad8f1c281b3620cd7691d"}
Sep 16 19:00:40.428233 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.428191    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k" event={"ID":"c896c97a-e86f-4f7b-9384-013e0da2a732","Type":"ContainerStarted","Data":"0a97677cc5eb8b4aede565e4b5522690c14d8ccbf18a43b8eaf2e7b05a5c506d"}
Sep 16 19:00:40.428763 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.428302    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:40.470600 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.470548    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k" podStartSLOduration=1.906478967 podStartE2EDuration="2.470532785s" totalImagesPullingTime="564.053818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:40.468603591 +0000 UTC m=+1268.285369178" watchObservedRunningTime="2026-09-16 19:00:40.470532785 +0000 UTC m=+1268.287298371"
Sep 16 19:00:40.471243 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.471225    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"]
Sep 16 19:00:40.474848 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.474829    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:40.478691 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.478670    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-96b42948\""
Sep 16 19:00:40.478838 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.478736    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-96b42948-dockercfg-cgrvq\""
Sep 16 19:00:40.500701 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.500666    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"]
Sep 16 19:00:40.596648 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.596612    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2wq\" (UniqueName: \"kubernetes.io/projected/051d5b9e-1897-458b-acc8-29b8b74250eb-kube-api-access-zs2wq\") pod \"payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj\" (UID: \"051d5b9e-1897-458b-acc8-29b8b74250eb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:40.596973 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.596946    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051d5b9e-1897-458b-acc8-29b8b74250eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj\" (UID: \"051d5b9e-1897-458b-acc8-29b8b74250eb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:40.652582 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.652545    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"]
Sep 16 19:00:40.656143 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.656122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:40.698625 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.698520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2wq\" (UniqueName: \"kubernetes.io/projected/051d5b9e-1897-458b-acc8-29b8b74250eb-kube-api-access-zs2wq\") pod \"payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj\" (UID: \"051d5b9e-1897-458b-acc8-29b8b74250eb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:40.698834 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.698756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051d5b9e-1897-458b-acc8-29b8b74250eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj\" (UID: \"051d5b9e-1897-458b-acc8-29b8b74250eb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:40.699821 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.699771    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051d5b9e-1897-458b-acc8-29b8b74250eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj\" (UID: \"051d5b9e-1897-458b-acc8-29b8b74250eb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:40.701920 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.701895    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"]
Sep 16 19:00:40.715862 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.715831    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2wq\" (UniqueName: \"kubernetes.io/projected/051d5b9e-1897-458b-acc8-29b8b74250eb-kube-api-access-zs2wq\") pod \"payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj\" (UID: \"051d5b9e-1897-458b-acc8-29b8b74250eb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:40.786875 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.786837    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:40.800336 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.800301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-plugins-config-volume\") pod \"payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7\" (UID: \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\") " pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:40.800467 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.800447    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwlt\" (UniqueName: \"kubernetes.io/projected/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-kube-api-access-nxwlt\") pod \"payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7\" (UID: \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\") " pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:40.901797 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.901766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-plugins-config-volume\") pod \"payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7\" (UID: \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\") " pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:40.901987 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.901843    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwlt\" (UniqueName: \"kubernetes.io/projected/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-kube-api-access-nxwlt\") pod \"payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7\" (UID: \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\") " pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:40.902474 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.902453    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-plugins-config-volume\") pod \"payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7\" (UID: \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\") " pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:40.914871 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.914842    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwlt\" (UniqueName: \"kubernetes.io/projected/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-kube-api-access-nxwlt\") pod \"payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7\" (UID: \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\") " pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:40.966227 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:40.966133    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:41.135153 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:41.135124    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"]
Sep 16 19:00:41.137026 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:00:41.136997    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051d5b9e_1897_458b_acc8_29b8b74250eb.slice/crio-3e071bd6be77981e23d5cf48992c6b7225c5b785b6b9eb53b9b00ba3bbb0886e WatchSource:0}: Error finding container 3e071bd6be77981e23d5cf48992c6b7225c5b785b6b9eb53b9b00ba3bbb0886e: Status 404 returned error can't find the container with id 3e071bd6be77981e23d5cf48992c6b7225c5b785b6b9eb53b9b00ba3bbb0886e
Sep 16 19:00:41.355932 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:41.354036    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"]
Sep 16 19:00:41.439732 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:41.439607    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7" event={"ID":"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c","Type":"ContainerStarted","Data":"5a2687d83f44b0ee781fe4145932e6c528778fdfdccc8c3453e067c637684f8a"}
Sep 16 19:00:41.439732 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:41.439679    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:41.439732 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:41.439703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7" event={"ID":"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c","Type":"ContainerStarted","Data":"6d66b25ee590be8cc7ee6b079115d1a27ebe82adec7722f06c643cded975d9f4"}
Sep 16 19:00:41.441477 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:41.441442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj" event={"ID":"051d5b9e-1897-458b-acc8-29b8b74250eb","Type":"ContainerStarted","Data":"9a0fca530bf75fb2f630661ef7e4551daed68e0adac7d8bf12093c69f146e1e3"}
Sep 16 19:00:41.441601 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:41.441505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj" event={"ID":"051d5b9e-1897-458b-acc8-29b8b74250eb","Type":"ContainerStarted","Data":"3e071bd6be77981e23d5cf48992c6b7225c5b785b6b9eb53b9b00ba3bbb0886e"}
Sep 16 19:00:41.469650 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:41.469595    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7" podStartSLOduration=1.46957873 podStartE2EDuration="1.46957873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:41.467820678 +0000 UTC m=+1269.284586265" watchObservedRunningTime="2026-09-16 19:00:41.46957873 +0000 UTC m=+1269.286344369"
Sep 16 19:00:41.513767 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:41.513683    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj" podStartSLOduration=1.5136610240000001 podStartE2EDuration="1.513661024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:41.507873723 +0000 UTC m=+1269.324639337" watchObservedRunningTime="2026-09-16 19:00:41.513661024 +0000 UTC m=+1269.330426610"
Sep 16 19:00:42.447085 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:42.447046    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:46.447413 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:46.447379    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:00:47.322530 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:47.322490    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:00:52.449549 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:52.449518    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:00:53.455523 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:53.455486    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:00:55.705152 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:55.705119    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"]
Sep 16 19:00:55.705515 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:55.705299    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr" podUID="d20a83d1-f8a4-4ef7-ae41-930ef65e3669" containerName="maas-api" containerID="cri-o://c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c" gracePeriod=30
Sep 16 19:00:55.969328 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:55.969260    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 19:00:56.057675 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.057635    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-metrics-certs\") pod \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") "
Sep 16 19:00:56.057913 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.057692    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-openshift-service-ca\") pod \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") "
Sep 16 19:00:56.057913 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.057749    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftc2t\" (UniqueName: \"kubernetes.io/projected/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-kube-api-access-ftc2t\") pod \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") "
Sep 16 19:00:56.057913 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.057842    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-maas-api-tls\") pod \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\" (UID: \"d20a83d1-f8a4-4ef7-ae41-930ef65e3669\") "
Sep 16 19:00:56.058168 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.058141    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d20a83d1-f8a4-4ef7-ae41-930ef65e3669" (UID: "d20a83d1-f8a4-4ef7-ae41-930ef65e3669"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:56.059897 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.059874    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d20a83d1-f8a4-4ef7-ae41-930ef65e3669" (UID: "d20a83d1-f8a4-4ef7-ae41-930ef65e3669"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:56.059979 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.059957    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-kube-api-access-ftc2t" (OuterVolumeSpecName: "kube-api-access-ftc2t") pod "d20a83d1-f8a4-4ef7-ae41-930ef65e3669" (UID: "d20a83d1-f8a4-4ef7-ae41-930ef65e3669"). InnerVolumeSpecName "kube-api-access-ftc2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:56.060015 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.059974    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d20a83d1-f8a4-4ef7-ae41-930ef65e3669" (UID: "d20a83d1-f8a4-4ef7-ae41-930ef65e3669"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:56.158782 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.158741    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:56.158782 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.158776    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:56.158782 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.158786    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftc2t\" (UniqueName: \"kubernetes.io/projected/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-kube-api-access-ftc2t\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:56.159021 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.158795    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20a83d1-f8a4-4ef7-ae41-930ef65e3669-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:00:56.526861 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.526821    2575 generic.go:356] "Generic (PLEG): container finished" podID="d20a83d1-f8a4-4ef7-ae41-930ef65e3669" containerID="c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c" exitCode=0
Sep 16 19:00:56.527029 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.526866    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"
Sep 16 19:00:56.527029 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.526907    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr" event={"ID":"d20a83d1-f8a4-4ef7-ae41-930ef65e3669","Type":"ContainerDied","Data":"c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c"}
Sep 16 19:00:56.527029 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.526958    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr" event={"ID":"d20a83d1-f8a4-4ef7-ae41-930ef65e3669","Type":"ContainerDied","Data":"a50130ff2057fc1be3b26422167367984fef62ec9a012886685559ee2d66ccaa"}
Sep 16 19:00:56.527029 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.526982    2575 scope.go:117] "RemoveContainer" containerID="c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c"
Sep 16 19:00:56.537308 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.537286    2575 scope.go:117] "RemoveContainer" containerID="c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c"
Sep 16 19:00:56.537573 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:00:56.537554    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c\": container with ID starting with c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c not found: ID does not exist" containerID="c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c"
Sep 16 19:00:56.537619 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.537583    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c"} err="failed to get container status \"c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c\": rpc error: code = NotFound desc = could not find container \"c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c\": container with ID starting with c6251daf59dc7737a350b137da663eb86c23f2a50f9815230600311bfac7963c not found: ID does not exist"
Sep 16 19:00:56.552407 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.552377    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"]
Sep 16 19:00:56.557435 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.557410    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2efa6f-648597db5f-j27lr"]
Sep 16 19:00:56.621115 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:00:56.621085    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20a83d1-f8a4-4ef7-ae41-930ef65e3669" path="/var/lib/kubelet/pods/d20a83d1-f8a4-4ef7-ae41-930ef65e3669/volumes"
Sep 16 19:01:06.290315 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.290274    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"]
Sep 16 19:01:06.290767 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.290506    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k" podUID="c896c97a-e86f-4f7b-9384-013e0da2a732" containerName="maas-api" containerID="cri-o://243f5cc0a6eb1549d92ef1a459747502473fa6b9655ad8f1c281b3620cd7691d" gracePeriod=30
Sep 16 19:01:06.362891 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.362847    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"]
Sep 16 19:01:06.363177 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.363147    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7" podUID="b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c" containerName="payload-processing" containerID="cri-o://5a2687d83f44b0ee781fe4145932e6c528778fdfdccc8c3453e067c637684f8a" gracePeriod=30
Sep 16 19:01:06.381824 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.381794    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"]
Sep 16 19:01:06.382061 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.382030    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj" podUID="051d5b9e-1897-458b-acc8-29b8b74250eb" containerName="payload-pre-processing" containerID="cri-o://9a0fca530bf75fb2f630661ef7e4551daed68e0adac7d8bf12093c69f146e1e3" gracePeriod=30
Sep 16 19:01:06.441940 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.441898    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k" podUID="c896c97a-e86f-4f7b-9384-013e0da2a732" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.55:8443/health\": dial tcp 10.134.0.55:8443: connect: connection refused"
Sep 16 19:01:06.584056 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.584013    2575 generic.go:356] "Generic (PLEG): container finished" podID="b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c" containerID="5a2687d83f44b0ee781fe4145932e6c528778fdfdccc8c3453e067c637684f8a" exitCode=0
Sep 16 19:01:06.584256 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.584082    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7" event={"ID":"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c","Type":"ContainerDied","Data":"5a2687d83f44b0ee781fe4145932e6c528778fdfdccc8c3453e067c637684f8a"}
Sep 16 19:01:06.585959 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.585937    2575 generic.go:356] "Generic (PLEG): container finished" podID="051d5b9e-1897-458b-acc8-29b8b74250eb" containerID="9a0fca530bf75fb2f630661ef7e4551daed68e0adac7d8bf12093c69f146e1e3" exitCode=0
Sep 16 19:01:06.586059 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.585971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj" event={"ID":"051d5b9e-1897-458b-acc8-29b8b74250eb","Type":"ContainerDied","Data":"9a0fca530bf75fb2f630661ef7e4551daed68e0adac7d8bf12093c69f146e1e3"}
Sep 16 19:01:06.587839 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.587819    2575 generic.go:356] "Generic (PLEG): container finished" podID="c896c97a-e86f-4f7b-9384-013e0da2a732" containerID="243f5cc0a6eb1549d92ef1a459747502473fa6b9655ad8f1c281b3620cd7691d" exitCode=0
Sep 16 19:01:06.587924 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.587864    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k" event={"ID":"c896c97a-e86f-4f7b-9384-013e0da2a732","Type":"ContainerDied","Data":"243f5cc0a6eb1549d92ef1a459747502473fa6b9655ad8f1c281b3620cd7691d"}
Sep 16 19:01:06.651390 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.651343    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:01:06.681677 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.681649    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:01:06.711451 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.711424    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:01:06.761305 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.761276    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxwlt\" (UniqueName: \"kubernetes.io/projected/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-kube-api-access-nxwlt\") pod \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\" (UID: \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\") "
Sep 16 19:01:06.761475 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.761322    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-plugins-config-volume\") pod \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\" (UID: \"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c\") "
Sep 16 19:01:06.761475 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.761372    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbb9k\" (UniqueName: \"kubernetes.io/projected/c896c97a-e86f-4f7b-9384-013e0da2a732-kube-api-access-bbb9k\") pod \"c896c97a-e86f-4f7b-9384-013e0da2a732\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") "
Sep 16 19:01:06.761475 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.761431    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-maas-api-tls\") pod \"c896c97a-e86f-4f7b-9384-013e0da2a732\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") "
Sep 16 19:01:06.761475 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.761467    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c896c97a-e86f-4f7b-9384-013e0da2a732-openshift-service-ca\") pod \"c896c97a-e86f-4f7b-9384-013e0da2a732\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") "
Sep 16 19:01:06.761699 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.761491    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs2wq\" (UniqueName: \"kubernetes.io/projected/051d5b9e-1897-458b-acc8-29b8b74250eb-kube-api-access-zs2wq\") pod \"051d5b9e-1897-458b-acc8-29b8b74250eb\" (UID: \"051d5b9e-1897-458b-acc8-29b8b74250eb\") "
Sep 16 19:01:06.761699 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.761546    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051d5b9e-1897-458b-acc8-29b8b74250eb-plugins-config-volume\") pod \"051d5b9e-1897-458b-acc8-29b8b74250eb\" (UID: \"051d5b9e-1897-458b-acc8-29b8b74250eb\") "
Sep 16 19:01:06.761699 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.761564    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-metrics-certs\") pod \"c896c97a-e86f-4f7b-9384-013e0da2a732\" (UID: \"c896c97a-e86f-4f7b-9384-013e0da2a732\") "
Sep 16 19:01:06.762309 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.762251    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c896c97a-e86f-4f7b-9384-013e0da2a732-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c896c97a-e86f-4f7b-9384-013e0da2a732" (UID: "c896c97a-e86f-4f7b-9384-013e0da2a732"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:06.762553 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.762319    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c" (UID: "b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:06.762768 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.762742    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051d5b9e-1897-458b-acc8-29b8b74250eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "051d5b9e-1897-458b-acc8-29b8b74250eb" (UID: "051d5b9e-1897-458b-acc8-29b8b74250eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:06.764302 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.764277    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c896c97a-e86f-4f7b-9384-013e0da2a732-kube-api-access-bbb9k" (OuterVolumeSpecName: "kube-api-access-bbb9k") pod "c896c97a-e86f-4f7b-9384-013e0da2a732" (UID: "c896c97a-e86f-4f7b-9384-013e0da2a732"). InnerVolumeSpecName "kube-api-access-bbb9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:06.764302 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.764292    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c896c97a-e86f-4f7b-9384-013e0da2a732" (UID: "c896c97a-e86f-4f7b-9384-013e0da2a732"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:06.764503 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.764470    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c896c97a-e86f-4f7b-9384-013e0da2a732" (UID: "c896c97a-e86f-4f7b-9384-013e0da2a732"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:06.764763 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.764733    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-kube-api-access-nxwlt" (OuterVolumeSpecName: "kube-api-access-nxwlt") pod "b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c" (UID: "b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c"). InnerVolumeSpecName "kube-api-access-nxwlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:06.765338 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.765321    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051d5b9e-1897-458b-acc8-29b8b74250eb-kube-api-access-zs2wq" (OuterVolumeSpecName: "kube-api-access-zs2wq") pod "051d5b9e-1897-458b-acc8-29b8b74250eb" (UID: "051d5b9e-1897-458b-acc8-29b8b74250eb"). InnerVolumeSpecName "kube-api-access-zs2wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:06.863302 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.863267    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:01:06.863302 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.863299    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c896c97a-e86f-4f7b-9384-013e0da2a732-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:01:06.863302 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.863310    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2wq\" (UniqueName: \"kubernetes.io/projected/051d5b9e-1897-458b-acc8-29b8b74250eb-kube-api-access-zs2wq\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:01:06.863543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.863319    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051d5b9e-1897-458b-acc8-29b8b74250eb-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:01:06.863543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.863328    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c896c97a-e86f-4f7b-9384-013e0da2a732-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:01:06.863543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.863336    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxwlt\" (UniqueName: \"kubernetes.io/projected/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-kube-api-access-nxwlt\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:01:06.863543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.863346    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:01:06.863543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:06.863354    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbb9k\" (UniqueName: \"kubernetes.io/projected/c896c97a-e86f-4f7b-9384-013e0da2a732-kube-api-access-bbb9k\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:01:07.595048 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.595017    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"
Sep 16 19:01:07.595528 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.595058    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj" event={"ID":"051d5b9e-1897-458b-acc8-29b8b74250eb","Type":"ContainerDied","Data":"3e071bd6be77981e23d5cf48992c6b7225c5b785b6b9eb53b9b00ba3bbb0886e"}
Sep 16 19:01:07.595528 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.595127    2575 scope.go:117] "RemoveContainer" containerID="9a0fca530bf75fb2f630661ef7e4551daed68e0adac7d8bf12093c69f146e1e3"
Sep 16 19:01:07.596544 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.596350    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"
Sep 16 19:01:07.596544 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.596392    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k" event={"ID":"c896c97a-e86f-4f7b-9384-013e0da2a732","Type":"ContainerDied","Data":"0a97677cc5eb8b4aede565e4b5522690c14d8ccbf18a43b8eaf2e7b05a5c506d"}
Sep 16 19:01:07.598005 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.597982    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7" event={"ID":"b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c","Type":"ContainerDied","Data":"6d66b25ee590be8cc7ee6b079115d1a27ebe82adec7722f06c643cded975d9f4"}
Sep 16 19:01:07.598125 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.598025    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"
Sep 16 19:01:07.606094 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.606068    2575 scope.go:117] "RemoveContainer" containerID="243f5cc0a6eb1549d92ef1a459747502473fa6b9655ad8f1c281b3620cd7691d"
Sep 16 19:01:07.616878 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.616856    2575 scope.go:117] "RemoveContainer" containerID="5a2687d83f44b0ee781fe4145932e6c528778fdfdccc8c3453e067c637684f8a"
Sep 16 19:01:07.655438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.655402    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"]
Sep 16 19:01:07.718483 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.718446    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96b42948-7f984b4b5-bhpz7"]
Sep 16 19:01:07.817592 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.817558    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"]
Sep 16 19:01:07.819639 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.819616    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96b42948-5ccf4577f4-vw5tj"]
Sep 16 19:01:07.870926 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.870853    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"]
Sep 16 19:01:07.887897 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:07.887869    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96b42948-5976996d77-qbz8k"]
Sep 16 19:01:08.438497 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:08.438462    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:01:08.621605 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:08.621571    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="051d5b9e-1897-458b-acc8-29b8b74250eb" path="/var/lib/kubelet/pods/051d5b9e-1897-458b-acc8-29b8b74250eb/volumes"
Sep 16 19:01:08.622024 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:08.622009    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c" path="/var/lib/kubelet/pods/b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c/volumes"
Sep 16 19:01:08.622370 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:08.622357    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c896c97a-e86f-4f7b-9384-013e0da2a732" path="/var/lib/kubelet/pods/c896c97a-e86f-4f7b-9384-013e0da2a732/volumes"
Sep 16 19:01:27.436510 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:27.436473    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:01:34.206744 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.206691    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"]
Sep 16 19:01:34.207267 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207250    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20a83d1-f8a4-4ef7-ae41-930ef65e3669" containerName="maas-api"
Sep 16 19:01:34.207315 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207269    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20a83d1-f8a4-4ef7-ae41-930ef65e3669" containerName="maas-api"
Sep 16 19:01:34.207315 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207286    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c896c97a-e86f-4f7b-9384-013e0da2a732" containerName="maas-api"
Sep 16 19:01:34.207315 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207292    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c896c97a-e86f-4f7b-9384-013e0da2a732" containerName="maas-api"
Sep 16 19:01:34.207315 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207301    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c" containerName="payload-processing"
Sep 16 19:01:34.207315 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207307    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c" containerName="payload-processing"
Sep 16 19:01:34.207474 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207340    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="051d5b9e-1897-458b-acc8-29b8b74250eb" containerName="payload-pre-processing"
Sep 16 19:01:34.207474 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207347    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="051d5b9e-1897-458b-acc8-29b8b74250eb" containerName="payload-pre-processing"
Sep 16 19:01:34.207474 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207406    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c896c97a-e86f-4f7b-9384-013e0da2a732" containerName="maas-api"
Sep 16 19:01:34.207474 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207415    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="051d5b9e-1897-458b-acc8-29b8b74250eb" containerName="payload-pre-processing"
Sep 16 19:01:34.207474 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207421    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b663ab3b-8c20-4edb-a3f4-2c22b83b6b5c" containerName="payload-processing"
Sep 16 19:01:34.207474 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.207429    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20a83d1-f8a4-4ef7-ae41-930ef65e3669" containerName="maas-api"
Sep 16 19:01:34.211950 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.211934    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.217373 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.217347    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"default-dockercfg-9h2xx\""
Sep 16 19:01:34.217816 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.217796    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"kube-root-ca.crt\""
Sep 16 19:01:34.217934 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.217840    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"premium-w4-c090c8-kserve-self-signed-certs\""
Sep 16 19:01:34.218098 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.218081    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"openshift-service-ca.crt\""
Sep 16 19:01:34.225735 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.225698    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"]
Sep 16 19:01:34.251113 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.251087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-model-cache\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.251230 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.251121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhzq\" (UniqueName: \"kubernetes.io/projected/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-kube-api-access-tzhzq\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.251230 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.251169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-home\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.251303 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.251271    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-dshm\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.251366 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.251304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tls-certs\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.251366 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.251331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tmp-dir\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.352029 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.351989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-dshm\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.352029 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.352034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tls-certs\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.352255 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.352061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tmp-dir\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.352255 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.352092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-model-cache\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.352255 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.352111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhzq\" (UniqueName: \"kubernetes.io/projected/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-kube-api-access-tzhzq\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.352255 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.352143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-home\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.352495 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.352458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-model-cache\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.352549 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.352503    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-home\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.352549 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.352525    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tmp-dir\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.354390 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.354370    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-dshm\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.354513 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.354497    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tls-certs\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.365578 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.365556    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhzq\" (UniqueName: \"kubernetes.io/projected/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-kube-api-access-tzhzq\") pod \"premium-w4-c090c8-kserve-6487f58547-2rbdt\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") " pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.521264 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.521195    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:34.800494 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:34.800471    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"]
Sep 16 19:01:34.802451 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:01:34.802425    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ed67a6f_8fbd_48c2_b8b4_70ae56ae86ce.slice/crio-7b77b9f44a51705f67d990d9114ca282927d54765b289191a19940ebb6d77200 WatchSource:0}: Error finding container 7b77b9f44a51705f67d990d9114ca282927d54765b289191a19940ebb6d77200: Status 404 returned error can't find the container with id 7b77b9f44a51705f67d990d9114ca282927d54765b289191a19940ebb6d77200
Sep 16 19:01:35.760131 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:35.760095    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt" event={"ID":"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce","Type":"ContainerStarted","Data":"62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d"}
Sep 16 19:01:35.760530 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:35.760139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt" event={"ID":"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce","Type":"ContainerStarted","Data":"7b77b9f44a51705f67d990d9114ca282927d54765b289191a19940ebb6d77200"}
Sep 16 19:01:35.779860 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:35.779579    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt" podStartSLOduration=1.779560668 podStartE2EDuration="1.779560668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:35.778382925 +0000 UTC m=+1323.595148513" watchObservedRunningTime="2026-09-16 19:01:35.779560668 +0000 UTC m=+1323.596326256"
Sep 16 19:01:41.735115 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:41.735075    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:01:44.522352 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:44.522308    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:44.522803 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:44.522586    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:44.535259 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:44.535222    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:44.820086 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:44.820006    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:01:49.201114 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.201074    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"]
Sep 16 19:01:49.205501 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.205479    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.209385 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.209362    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-9d2450-serving-cert\""
Sep 16 19:01:49.224767 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.224725    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"]
Sep 16 19:01:49.305740 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.305676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk5d9\" (UniqueName: \"kubernetes.io/projected/a1f962d8-4e78-401b-ba64-5b82a0de29c3-kube-api-access-jk5d9\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.305933 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.305755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f962d8-4e78-401b-ba64-5b82a0de29c3-openshift-service-ca\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.305933 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.305805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.306026 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.305946    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-metrics-certs\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.407511 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.407464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-metrics-certs\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.407681 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.407574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5d9\" (UniqueName: \"kubernetes.io/projected/a1f962d8-4e78-401b-ba64-5b82a0de29c3-kube-api-access-jk5d9\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.407681 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.407605    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f962d8-4e78-401b-ba64-5b82a0de29c3-openshift-service-ca\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.407681 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.407650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.407905 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:01:49.407827    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-serving-cert: secret "maas-api-e2e-explicit-9d2450-serving-cert" not found
Sep 16 19:01:49.407969 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:01:49.407913    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls podName:a1f962d8-4e78-401b-ba64-5b82a0de29c3 nodeName:}" failed. No retries permitted until 2026-09-16 19:01:49.907891344 +0000 UTC m=+1337.724656929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls") pod "maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs" (UID: "a1f962d8-4e78-401b-ba64-5b82a0de29c3") : secret "maas-api-e2e-explicit-9d2450-serving-cert" not found
Sep 16 19:01:49.408360 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.408333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f962d8-4e78-401b-ba64-5b82a0de29c3-openshift-service-ca\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.410224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.410197    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-metrics-certs\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.419087 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.419064    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5d9\" (UniqueName: \"kubernetes.io/projected/a1f962d8-4e78-401b-ba64-5b82a0de29c3-kube-api-access-jk5d9\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.912544 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:49.912476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:49.912795 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:01:49.912651    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-serving-cert: secret "maas-api-e2e-explicit-9d2450-serving-cert" not found
Sep 16 19:01:49.912795 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:01:49.912771    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls podName:a1f962d8-4e78-401b-ba64-5b82a0de29c3 nodeName:}" failed. No retries permitted until 2026-09-16 19:01:50.912751061 +0000 UTC m=+1338.729516625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls") pod "maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs" (UID: "a1f962d8-4e78-401b-ba64-5b82a0de29c3") : secret "maas-api-e2e-explicit-9d2450-serving-cert" not found
Sep 16 19:01:50.061580 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.061537    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"]
Sep 16 19:01:50.065477 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.065451    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:01:50.069307 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.069279    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9d2450\""
Sep 16 19:01:50.069920 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.069898    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-9d2450-dockercfg-bsrzl\""
Sep 16 19:01:50.078316 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.078292    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"]
Sep 16 19:01:50.216756 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.216626    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca0c30b-11b4-464d-b28c-939e534e39f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l\" (UID: \"9ca0c30b-11b4-464d-b28c-939e534e39f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:01:50.216756 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.216735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gg8j\" (UniqueName: \"kubernetes.io/projected/9ca0c30b-11b4-464d-b28c-939e534e39f1-kube-api-access-2gg8j\") pod \"payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l\" (UID: \"9ca0c30b-11b4-464d-b28c-939e534e39f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:01:50.317590 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.317544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gg8j\" (UniqueName: \"kubernetes.io/projected/9ca0c30b-11b4-464d-b28c-939e534e39f1-kube-api-access-2gg8j\") pod \"payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l\" (UID: \"9ca0c30b-11b4-464d-b28c-939e534e39f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:01:50.317788 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.317681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca0c30b-11b4-464d-b28c-939e534e39f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l\" (UID: \"9ca0c30b-11b4-464d-b28c-939e534e39f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:01:50.318547 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.318523    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca0c30b-11b4-464d-b28c-939e534e39f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l\" (UID: \"9ca0c30b-11b4-464d-b28c-939e534e39f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:01:50.326761 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.326731    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gg8j\" (UniqueName: \"kubernetes.io/projected/9ca0c30b-11b4-464d-b28c-939e534e39f1-kube-api-access-2gg8j\") pod \"payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l\" (UID: \"9ca0c30b-11b4-464d-b28c-939e534e39f1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:01:50.377455 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.377412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:01:50.670602 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.670559    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"]
Sep 16 19:01:50.673059 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:01:50.673030    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca0c30b_11b4_464d_b28c_939e534e39f1.slice/crio-4e518a21110ff6a2fed71faf54c8239d76cd3e27ec0612db967561543bbcec23 WatchSource:0}: Error finding container 4e518a21110ff6a2fed71faf54c8239d76cd3e27ec0612db967561543bbcec23: Status 404 returned error can't find the container with id 4e518a21110ff6a2fed71faf54c8239d76cd3e27ec0612db967561543bbcec23
Sep 16 19:01:50.843445 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.843394    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l" event={"ID":"9ca0c30b-11b4-464d-b28c-939e534e39f1","Type":"ContainerStarted","Data":"8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb"}
Sep 16 19:01:50.843618 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.843463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l" event={"ID":"9ca0c30b-11b4-464d-b28c-939e534e39f1","Type":"ContainerStarted","Data":"4e518a21110ff6a2fed71faf54c8239d76cd3e27ec0612db967561543bbcec23"}
Sep 16 19:01:50.843618 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.843601    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:01:50.866306 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.866253    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l" podStartSLOduration=0.866236407 podStartE2EDuration="866.236407ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:50.863775504 +0000 UTC m=+1338.680541095" watchObservedRunningTime="2026-09-16 19:01:50.866236407 +0000 UTC m=+1338.683001992"
Sep 16 19:01:50.923465 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.923369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:50.925933 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:50.925903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls\") pod \"maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:51.018291 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:51.018249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:51.314587 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:51.314555    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"]
Sep 16 19:01:51.315680 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:01:51.315653    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f962d8_4e78_401b_ba64_5b82a0de29c3.slice/crio-7babf8da1b0bdc8a369da633082351116621794eb72157a0b5b8c55fa5302ba1 WatchSource:0}: Error finding container 7babf8da1b0bdc8a369da633082351116621794eb72157a0b5b8c55fa5302ba1: Status 404 returned error can't find the container with id 7babf8da1b0bdc8a369da633082351116621794eb72157a0b5b8c55fa5302ba1
Sep 16 19:01:51.851902 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:51.851866    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs" event={"ID":"a1f962d8-4e78-401b-ba64-5b82a0de29c3","Type":"ContainerStarted","Data":"7babf8da1b0bdc8a369da633082351116621794eb72157a0b5b8c55fa5302ba1"}
Sep 16 19:01:52.859811 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:52.859775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs" event={"ID":"a1f962d8-4e78-401b-ba64-5b82a0de29c3","Type":"ContainerStarted","Data":"362dd983de0ef2e569a0d9ea03ded979da059f87c34723d383825179c4366a68"}
Sep 16 19:01:52.860230 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:52.859969    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:01:52.880419 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:52.880363    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs" podStartSLOduration=3.087055533 podStartE2EDuration="3.880348776s" totalImagesPullingTime="793.293243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:52.877603354 +0000 UTC m=+1340.694368940" watchObservedRunningTime="2026-09-16 19:01:52.880348776 +0000 UTC m=+1340.697114363"
Sep 16 19:01:58.509901 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:58.509819    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:01:58.872244 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:01:58.872215    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:02:01.854532 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:02:01.854492    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:02:25.341842 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:02:25.341800    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:02:45.013907 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:02:45.013864    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:03:02.710348 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.710261    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"]
Sep 16 19:03:02.714230 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.714210    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.717040 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.717017    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-050f3bde-serving-cert\""
Sep 16 19:03:02.728441 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.728401    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"]
Sep 16 19:03:02.868252 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.868208    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3307ed7-3590-4952-b55a-f6eda4bafcd6-openshift-service-ca\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.868449 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.868312    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-maas-api-tls\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.868449 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.868370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hk5\" (UniqueName: \"kubernetes.io/projected/b3307ed7-3590-4952-b55a-f6eda4bafcd6-kube-api-access-q8hk5\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.868554 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.868466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-metrics-certs\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.969196 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.969099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-maas-api-tls\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.969196 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.969160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hk5\" (UniqueName: \"kubernetes.io/projected/b3307ed7-3590-4952-b55a-f6eda4bafcd6-kube-api-access-q8hk5\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.969431 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.969236    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-metrics-certs\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.969431 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.969325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3307ed7-3590-4952-b55a-f6eda4bafcd6-openshift-service-ca\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.970082 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.970050    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3307ed7-3590-4952-b55a-f6eda4bafcd6-openshift-service-ca\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.972049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.972020    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-metrics-certs\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.972049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.972037    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-maas-api-tls\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:02.986982 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:02.986950    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hk5\" (UniqueName: \"kubernetes.io/projected/b3307ed7-3590-4952-b55a-f6eda4bafcd6-kube-api-access-q8hk5\") pod \"maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:03.025817 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.025772    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:03.303777 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.303742    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"]
Sep 16 19:03:03.306131 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:03:03.306098    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3307ed7_3590_4952_b55a_f6eda4bafcd6.slice/crio-0f592d4a863256266456503ca156f4105076422887525eddfc03c93b2f1443c8 WatchSource:0}: Error finding container 0f592d4a863256266456503ca156f4105076422887525eddfc03c93b2f1443c8: Status 404 returned error can't find the container with id 0f592d4a863256266456503ca156f4105076422887525eddfc03c93b2f1443c8
Sep 16 19:03:03.682768 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.682728    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"]
Sep 16 19:03:03.687921 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.687895    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:03.691738 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.691697    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-050f3bde\""
Sep 16 19:03:03.691917 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.691895    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-050f3bde-dockercfg-fp4nx\""
Sep 16 19:03:03.702777 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.702746    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"]
Sep 16 19:03:03.776665 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.776631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56\" (UID: \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:03.777083 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.776671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29v5t\" (UniqueName: \"kubernetes.io/projected/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-kube-api-access-29v5t\") pod \"payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56\" (UID: \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:03.878058 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.878025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56\" (UID: \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:03.878217 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.878070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29v5t\" (UniqueName: \"kubernetes.io/projected/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-kube-api-access-29v5t\") pod \"payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56\" (UID: \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:03.878809 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.878783    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56\" (UID: \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:03.889272 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.889245    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29v5t\" (UniqueName: \"kubernetes.io/projected/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-kube-api-access-29v5t\") pod \"payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56\" (UID: \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:03.999346 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:03.999249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:04.277312 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:04.277218    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh" event={"ID":"b3307ed7-3590-4952-b55a-f6eda4bafcd6","Type":"ContainerStarted","Data":"9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99"}
Sep 16 19:03:04.277312 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:04.277274    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh" event={"ID":"b3307ed7-3590-4952-b55a-f6eda4bafcd6","Type":"ContainerStarted","Data":"0f592d4a863256266456503ca156f4105076422887525eddfc03c93b2f1443c8"}
Sep 16 19:03:04.277515 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:04.277390    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:04.293882 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:04.293845    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"]
Sep 16 19:03:04.295438 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:03:04.295406    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cd519d_ac6c_40dd_9880_a4dcdf2882fc.slice/crio-17dccd83f3cdbd54e416b00d1b248dce551ae71c7e52ebde7bb12fad94278d15 WatchSource:0}: Error finding container 17dccd83f3cdbd54e416b00d1b248dce551ae71c7e52ebde7bb12fad94278d15: Status 404 returned error can't find the container with id 17dccd83f3cdbd54e416b00d1b248dce551ae71c7e52ebde7bb12fad94278d15
Sep 16 19:03:05.284900 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:05.284861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56" event={"ID":"85cd519d-ac6c-40dd-9880-a4dcdf2882fc","Type":"ContainerStarted","Data":"ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851"}
Sep 16 19:03:05.285292 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:05.284918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56" event={"ID":"85cd519d-ac6c-40dd-9880-a4dcdf2882fc","Type":"ContainerStarted","Data":"17dccd83f3cdbd54e416b00d1b248dce551ae71c7e52ebde7bb12fad94278d15"}
Sep 16 19:03:05.313879 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:05.313825    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56" podStartSLOduration=2.31380477 podStartE2EDuration="2.31380477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:05.311367181 +0000 UTC m=+1413.128132768" watchObservedRunningTime="2026-09-16 19:03:05.31380477 +0000 UTC m=+1413.130570356"
Sep 16 19:03:05.314248 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:05.314227    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh" podStartSLOduration=2.805206588 podStartE2EDuration="3.314221529s" totalImagesPullingTime="509.014941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:04.333155382 +0000 UTC m=+1412.149920969" watchObservedRunningTime="2026-09-16 19:03:05.314221529 +0000 UTC m=+1413.130987152"
Sep 16 19:03:06.290858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:06.290825    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:10.290936 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:10.290900    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:17.297873 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:17.297837    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:18.313224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:18.313189    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:03:47.628692 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:47.628660    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"]
Sep 16 19:03:47.629258 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:47.628841    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh" podUID="b3307ed7-3590-4952-b55a-f6eda4bafcd6" containerName="maas-api" containerID="cri-o://9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99" gracePeriod=30
Sep 16 19:03:47.701874 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:47.701840    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"]
Sep 16 19:03:47.702116 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:47.702085    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56" podUID="85cd519d-ac6c-40dd-9880-a4dcdf2882fc" containerName="payload-pre-processing" containerID="cri-o://ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851" gracePeriod=30
Sep 16 19:03:47.965332 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:47.965306    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:48.003681 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.003659    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:48.047214 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.047186    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8hk5\" (UniqueName: \"kubernetes.io/projected/b3307ed7-3590-4952-b55a-f6eda4bafcd6-kube-api-access-q8hk5\") pod \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") "
Sep 16 19:03:48.047395 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.047252    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29v5t\" (UniqueName: \"kubernetes.io/projected/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-kube-api-access-29v5t\") pod \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\" (UID: \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\") "
Sep 16 19:03:48.047395 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.047301    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-metrics-certs\") pod \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") "
Sep 16 19:03:48.047395 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.047371    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3307ed7-3590-4952-b55a-f6eda4bafcd6-openshift-service-ca\") pod \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") "
Sep 16 19:03:48.047552 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.047427    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-maas-api-tls\") pod \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\" (UID: \"b3307ed7-3590-4952-b55a-f6eda4bafcd6\") "
Sep 16 19:03:48.047552 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.047455    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-plugins-config-volume\") pod \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\" (UID: \"85cd519d-ac6c-40dd-9880-a4dcdf2882fc\") "
Sep 16 19:03:48.047858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.047826    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3307ed7-3590-4952-b55a-f6eda4bafcd6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3307ed7-3590-4952-b55a-f6eda4bafcd6" (UID: "b3307ed7-3590-4952-b55a-f6eda4bafcd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:48.048179 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.048135    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85cd519d-ac6c-40dd-9880-a4dcdf2882fc" (UID: "85cd519d-ac6c-40dd-9880-a4dcdf2882fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:48.049744 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.049698    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-kube-api-access-29v5t" (OuterVolumeSpecName: "kube-api-access-29v5t") pod "85cd519d-ac6c-40dd-9880-a4dcdf2882fc" (UID: "85cd519d-ac6c-40dd-9880-a4dcdf2882fc"). InnerVolumeSpecName "kube-api-access-29v5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:48.050027 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.050003    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3307ed7-3590-4952-b55a-f6eda4bafcd6" (UID: "b3307ed7-3590-4952-b55a-f6eda4bafcd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:48.050102 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.050044    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3307ed7-3590-4952-b55a-f6eda4bafcd6-kube-api-access-q8hk5" (OuterVolumeSpecName: "kube-api-access-q8hk5") pod "b3307ed7-3590-4952-b55a-f6eda4bafcd6" (UID: "b3307ed7-3590-4952-b55a-f6eda4bafcd6"). InnerVolumeSpecName "kube-api-access-q8hk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:48.050215 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.050191    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b3307ed7-3590-4952-b55a-f6eda4bafcd6" (UID: "b3307ed7-3590-4952-b55a-f6eda4bafcd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:48.148892 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.148859    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:03:48.148892 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.148889    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3307ed7-3590-4952-b55a-f6eda4bafcd6-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:03:48.149090 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.148903    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3307ed7-3590-4952-b55a-f6eda4bafcd6-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:03:48.149090 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.148916    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:03:48.149090 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.148930    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8hk5\" (UniqueName: \"kubernetes.io/projected/b3307ed7-3590-4952-b55a-f6eda4bafcd6-kube-api-access-q8hk5\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:03:48.149090 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.148941    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29v5t\" (UniqueName: \"kubernetes.io/projected/85cd519d-ac6c-40dd-9880-a4dcdf2882fc-kube-api-access-29v5t\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:03:48.526447 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.526413    2575 generic.go:356] "Generic (PLEG): container finished" podID="b3307ed7-3590-4952-b55a-f6eda4bafcd6" containerID="9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99" exitCode=0
Sep 16 19:03:48.526637 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.526454    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"
Sep 16 19:03:48.526637 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.526450    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh" event={"ID":"b3307ed7-3590-4952-b55a-f6eda4bafcd6","Type":"ContainerDied","Data":"9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99"}
Sep 16 19:03:48.526637 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.526509    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh" event={"ID":"b3307ed7-3590-4952-b55a-f6eda4bafcd6","Type":"ContainerDied","Data":"0f592d4a863256266456503ca156f4105076422887525eddfc03c93b2f1443c8"}
Sep 16 19:03:48.526637 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.526541    2575 scope.go:117] "RemoveContainer" containerID="9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99"
Sep 16 19:03:48.527880 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.527859    2575 generic.go:356] "Generic (PLEG): container finished" podID="85cd519d-ac6c-40dd-9880-a4dcdf2882fc" containerID="ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851" exitCode=0
Sep 16 19:03:48.527991 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.527899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56" event={"ID":"85cd519d-ac6c-40dd-9880-a4dcdf2882fc","Type":"ContainerDied","Data":"ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851"}
Sep 16 19:03:48.527991 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.527942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56" event={"ID":"85cd519d-ac6c-40dd-9880-a4dcdf2882fc","Type":"ContainerDied","Data":"17dccd83f3cdbd54e416b00d1b248dce551ae71c7e52ebde7bb12fad94278d15"}
Sep 16 19:03:48.527991 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.527904    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"
Sep 16 19:03:48.537524 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.537507    2575 scope.go:117] "RemoveContainer" containerID="9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99"
Sep 16 19:03:48.537896 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:03:48.537869    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99\": container with ID starting with 9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99 not found: ID does not exist" containerID="9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99"
Sep 16 19:03:48.537996 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.537909    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99"} err="failed to get container status \"9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99\": rpc error: code = NotFound desc = could not find container \"9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99\": container with ID starting with 9e74ad903119ae07379398693a4f1c07665081487346d6576ba9f56fc66ebf99 not found: ID does not exist"
Sep 16 19:03:48.537996 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.537932    2575 scope.go:117] "RemoveContainer" containerID="ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851"
Sep 16 19:03:48.546220 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.546203    2575 scope.go:117] "RemoveContainer" containerID="ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851"
Sep 16 19:03:48.546481 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:03:48.546465    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851\": container with ID starting with ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851 not found: ID does not exist" containerID="ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851"
Sep 16 19:03:48.546530 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.546486    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851"} err="failed to get container status \"ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851\": rpc error: code = NotFound desc = could not find container \"ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851\": container with ID starting with ac4d9e2ee096802dd091b9489a630808d8969c8694cd93351fc4cc88928f1851 not found: ID does not exist"
Sep 16 19:03:48.551189 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.551166    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"]
Sep 16 19:03:48.556515 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.556493    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-050f3bde-76597c8fc8-rnnfh"]
Sep 16 19:03:48.566950 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.566931    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"]
Sep 16 19:03:48.569261 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.569242    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-050f3bde-7d7f9f9b78-f4s56"]
Sep 16 19:03:48.621920 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.621889    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85cd519d-ac6c-40dd-9880-a4dcdf2882fc" path="/var/lib/kubelet/pods/85cd519d-ac6c-40dd-9880-a4dcdf2882fc/volumes"
Sep 16 19:03:48.622289 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:48.622276    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3307ed7-3590-4952-b55a-f6eda4bafcd6" path="/var/lib/kubelet/pods/b3307ed7-3590-4952-b55a-f6eda4bafcd6/volumes"
Sep 16 19:03:54.315626 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:03:54.315583    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:04:15.258067 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.258032    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"]
Sep 16 19:04:15.258513 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.258500    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3307ed7-3590-4952-b55a-f6eda4bafcd6" containerName="maas-api"
Sep 16 19:04:15.258556 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.258515    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3307ed7-3590-4952-b55a-f6eda4bafcd6" containerName="maas-api"
Sep 16 19:04:15.258556 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.258524    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85cd519d-ac6c-40dd-9880-a4dcdf2882fc" containerName="payload-pre-processing"
Sep 16 19:04:15.258556 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.258529    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="85cd519d-ac6c-40dd-9880-a4dcdf2882fc" containerName="payload-pre-processing"
Sep 16 19:04:15.258656 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.258626    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="85cd519d-ac6c-40dd-9880-a4dcdf2882fc" containerName="payload-pre-processing"
Sep 16 19:04:15.258656 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.258638    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3307ed7-3590-4952-b55a-f6eda4bafcd6" containerName="maas-api"
Sep 16 19:04:15.261973 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.261953    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.264443 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.264423    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"e2e-conflict-a-05590707-kserve-self-signed-certs\""
Sep 16 19:04:15.273605 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.273579    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"]
Sep 16 19:04:15.323810 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.323769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-dshm\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.323991 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.323817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-model-cache\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.323991 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.323878    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-tmp-dir\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.323991 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.323898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-home\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.323991 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.323960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltxg7\" (UniqueName: \"kubernetes.io/projected/859538bc-95a2-41a6-9e35-bb4fc9664338-kube-api-access-ltxg7\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.324129 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.324020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859538bc-95a2-41a6-9e35-bb4fc9664338-tls-certs\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.425593 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.425546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859538bc-95a2-41a6-9e35-bb4fc9664338-tls-certs\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.425828 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.425657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-dshm\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.425828 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.425691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-model-cache\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.425828 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.425767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-tmp-dir\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.425828 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.425799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-home\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.426039 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.425942    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltxg7\" (UniqueName: \"kubernetes.io/projected/859538bc-95a2-41a6-9e35-bb4fc9664338-kube-api-access-ltxg7\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.426214 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.426188    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-model-cache\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.426288 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.426220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-tmp-dir\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.426288 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.426249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-home\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.428084 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.428063    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-dshm\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.428231 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.428211    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859538bc-95a2-41a6-9e35-bb4fc9664338-tls-certs\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.435162 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.435134    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltxg7\" (UniqueName: \"kubernetes.io/projected/859538bc-95a2-41a6-9e35-bb4fc9664338-kube-api-access-ltxg7\") pod \"e2e-conflict-a-05590707-kserve-85d75df898-xkwpw\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") " pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.571608 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.571490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:15.853589 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:15.851599    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"]
Sep 16 19:04:16.684917 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:16.684878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw" event={"ID":"859538bc-95a2-41a6-9e35-bb4fc9664338","Type":"ContainerStarted","Data":"04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a"}
Sep 16 19:04:16.685310 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:16.684925    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw" event={"ID":"859538bc-95a2-41a6-9e35-bb4fc9664338","Type":"ContainerStarted","Data":"610f9b9b7facf9bae67124a76e307499574e72c43d127efdb18b5c33c5e45085"}
Sep 16 19:04:16.706289 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:16.706239    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw" podStartSLOduration=1.706223478 podStartE2EDuration="1.706223478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:16.704836262 +0000 UTC m=+1484.521601849" watchObservedRunningTime="2026-09-16 19:04:16.706223478 +0000 UTC m=+1484.522989064"
Sep 16 19:04:19.095503 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.095467    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"]
Sep 16 19:04:19.099084 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.099063    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.101474 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.101449    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c20219-serving-cert\""
Sep 16 19:04:19.109273 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.109193    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"]
Sep 16 19:04:19.164660 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.164633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tcw\" (UniqueName: \"kubernetes.io/projected/3c44466b-d290-4ae9-88c4-38797cf872da-kube-api-access-w6tcw\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.164805 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.164667    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.164860 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.164813    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.164860 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.164850    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c44466b-d290-4ae9-88c4-38797cf872da-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.265746 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.265698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.265877 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.265755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c44466b-d290-4ae9-88c4-38797cf872da-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.265877 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.265823    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tcw\" (UniqueName: \"kubernetes.io/projected/3c44466b-d290-4ae9-88c4-38797cf872da-kube-api-access-w6tcw\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.265877 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.265845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.266419 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.266394    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c44466b-d290-4ae9-88c4-38797cf872da-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.268067 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.268039    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.268149 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.268093    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.274165 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.274143    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tcw\" (UniqueName: \"kubernetes.io/projected/3c44466b-d290-4ae9-88c4-38797cf872da-kube-api-access-w6tcw\") pod \"maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.413993 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.413924    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:19.691911 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.691883    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"]
Sep 16 19:04:19.693744 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:04:19.693688    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c44466b_d290_4ae9_88c4_38797cf872da.slice/crio-40a18150ec8bbf8a570bccc58c791e776bee66784da4b1da0091e6b80c0d761d WatchSource:0}: Error finding container 40a18150ec8bbf8a570bccc58c791e776bee66784da4b1da0091e6b80c0d761d: Status 404 returned error can't find the container with id 40a18150ec8bbf8a570bccc58c791e776bee66784da4b1da0091e6b80c0d761d
Sep 16 19:04:19.695201 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.695180    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:04:19.704527 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.704501    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4" event={"ID":"3c44466b-d290-4ae9-88c4-38797cf872da","Type":"ContainerStarted","Data":"40a18150ec8bbf8a570bccc58c791e776bee66784da4b1da0091e6b80c0d761d"}
Sep 16 19:04:19.733023 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.732969    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"]
Sep 16 19:04:19.737875 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.737855    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:19.741500 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.741476    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c20219-dockercfg-5qb6z\""
Sep 16 19:04:19.741500 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.741494    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c20219\""
Sep 16 19:04:19.751191 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.751163    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"]
Sep 16 19:04:19.771136 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.771058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfsf\" (UniqueName: \"kubernetes.io/projected/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-kube-api-access-xgfsf\") pod \"payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv\" (UID: \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:19.771136 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.771118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv\" (UID: \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:19.872190 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.872158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfsf\" (UniqueName: \"kubernetes.io/projected/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-kube-api-access-xgfsf\") pod \"payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv\" (UID: \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:19.872190 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.872199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv\" (UID: \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:19.872963 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.872939    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv\" (UID: \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:19.881016 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:19.880990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfsf\" (UniqueName: \"kubernetes.io/projected/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-kube-api-access-xgfsf\") pod \"payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv\" (UID: \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:20.050860 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:20.050776    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:20.325828 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:20.325798    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"]
Sep 16 19:04:20.327699 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:04:20.327674    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b98ec98_39eb_4ed5_8ae1_641cf6093c9e.slice/crio-1be982908b5b5143807938e8234c1aa816c9fe25409e10f4facfe22e4662e909 WatchSource:0}: Error finding container 1be982908b5b5143807938e8234c1aa816c9fe25409e10f4facfe22e4662e909: Status 404 returned error can't find the container with id 1be982908b5b5143807938e8234c1aa816c9fe25409e10f4facfe22e4662e909
Sep 16 19:04:20.712362 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:20.712319    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv" event={"ID":"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e","Type":"ContainerStarted","Data":"0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8"}
Sep 16 19:04:20.712572 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:20.712388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv" event={"ID":"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e","Type":"ContainerStarted","Data":"1be982908b5b5143807938e8234c1aa816c9fe25409e10f4facfe22e4662e909"}
Sep 16 19:04:20.712572 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:20.712511    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:20.713959 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:20.713933    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4" event={"ID":"3c44466b-d290-4ae9-88c4-38797cf872da","Type":"ContainerStarted","Data":"113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9"}
Sep 16 19:04:20.714096 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:20.714076    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:20.736254 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:20.736190    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv" podStartSLOduration=1.736169729 podStartE2EDuration="1.736169729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:20.733192725 +0000 UTC m=+1488.549958311" watchObservedRunningTime="2026-09-16 19:04:20.736169729 +0000 UTC m=+1488.552935317"
Sep 16 19:04:20.761739 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:20.761418    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4" podStartSLOduration=1.3276069289999999 podStartE2EDuration="1.761393153s" totalImagesPullingTime="433.786224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:20.761236432 +0000 UTC m=+1488.578002032" watchObservedRunningTime="2026-09-16 19:04:20.761393153 +0000 UTC m=+1488.578158745"
Sep 16 19:04:24.424547 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:24.424513    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:04:24.564769 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:24.564732    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"]
Sep 16 19:04:24.565005 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:24.564977    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs" podUID="a1f962d8-4e78-401b-ba64-5b82a0de29c3" containerName="maas-api" containerID="cri-o://362dd983de0ef2e569a0d9ea03ded979da059f87c34723d383825179c4366a68" gracePeriod=30
Sep 16 19:04:24.646586 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:24.646552    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"]
Sep 16 19:04:24.646831 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:24.646803    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l" podUID="9ca0c30b-11b4-464d-b28c-939e534e39f1" containerName="payload-pre-processing" containerID="cri-o://8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb" gracePeriod=30
Sep 16 19:04:24.745697 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:24.745623    2575 generic.go:356] "Generic (PLEG): container finished" podID="a1f962d8-4e78-401b-ba64-5b82a0de29c3" containerID="362dd983de0ef2e569a0d9ea03ded979da059f87c34723d383825179c4366a68" exitCode=0
Sep 16 19:04:24.745844 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:24.745698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs" event={"ID":"a1f962d8-4e78-401b-ba64-5b82a0de29c3","Type":"ContainerDied","Data":"362dd983de0ef2e569a0d9ea03ded979da059f87c34723d383825179c4366a68"}
Sep 16 19:04:24.912522 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:24.912493    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:04:24.941269 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:24.941237    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:04:25.025357 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.025267    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gg8j\" (UniqueName: \"kubernetes.io/projected/9ca0c30b-11b4-464d-b28c-939e534e39f1-kube-api-access-2gg8j\") pod \"9ca0c30b-11b4-464d-b28c-939e534e39f1\" (UID: \"9ca0c30b-11b4-464d-b28c-939e534e39f1\") "
Sep 16 19:04:25.025357 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.025312    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk5d9\" (UniqueName: \"kubernetes.io/projected/a1f962d8-4e78-401b-ba64-5b82a0de29c3-kube-api-access-jk5d9\") pod \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") "
Sep 16 19:04:25.025357 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.025353    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f962d8-4e78-401b-ba64-5b82a0de29c3-openshift-service-ca\") pod \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") "
Sep 16 19:04:25.025637 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.025401    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca0c30b-11b4-464d-b28c-939e534e39f1-plugins-config-volume\") pod \"9ca0c30b-11b4-464d-b28c-939e534e39f1\" (UID: \"9ca0c30b-11b4-464d-b28c-939e534e39f1\") "
Sep 16 19:04:25.025637 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.025540    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-metrics-certs\") pod \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") "
Sep 16 19:04:25.025637 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.025593    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls\") pod \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\" (UID: \"a1f962d8-4e78-401b-ba64-5b82a0de29c3\") "
Sep 16 19:04:25.025814 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.025740    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f962d8-4e78-401b-ba64-5b82a0de29c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1f962d8-4e78-401b-ba64-5b82a0de29c3" (UID: "a1f962d8-4e78-401b-ba64-5b82a0de29c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:25.025869 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.025825    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca0c30b-11b4-464d-b28c-939e534e39f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ca0c30b-11b4-464d-b28c-939e534e39f1" (UID: "9ca0c30b-11b4-464d-b28c-939e534e39f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:25.026272 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.026218    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f962d8-4e78-401b-ba64-5b82a0de29c3-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:04:25.026272 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.026246    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca0c30b-11b4-464d-b28c-939e534e39f1-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:04:25.027648 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.027622    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f962d8-4e78-401b-ba64-5b82a0de29c3-kube-api-access-jk5d9" (OuterVolumeSpecName: "kube-api-access-jk5d9") pod "a1f962d8-4e78-401b-ba64-5b82a0de29c3" (UID: "a1f962d8-4e78-401b-ba64-5b82a0de29c3"). InnerVolumeSpecName "kube-api-access-jk5d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:25.027788 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.027687    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a1f962d8-4e78-401b-ba64-5b82a0de29c3" (UID: "a1f962d8-4e78-401b-ba64-5b82a0de29c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:25.028012 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.027997    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca0c30b-11b4-464d-b28c-939e534e39f1-kube-api-access-2gg8j" (OuterVolumeSpecName: "kube-api-access-2gg8j") pod "9ca0c30b-11b4-464d-b28c-939e534e39f1" (UID: "9ca0c30b-11b4-464d-b28c-939e534e39f1"). InnerVolumeSpecName "kube-api-access-2gg8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:25.028151 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.028138    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1f962d8-4e78-401b-ba64-5b82a0de29c3" (UID: "a1f962d8-4e78-401b-ba64-5b82a0de29c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:25.127176 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.127144    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gg8j\" (UniqueName: \"kubernetes.io/projected/9ca0c30b-11b4-464d-b28c-939e534e39f1-kube-api-access-2gg8j\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:04:25.127176 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.127172    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk5d9\" (UniqueName: \"kubernetes.io/projected/a1f962d8-4e78-401b-ba64-5b82a0de29c3-kube-api-access-jk5d9\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:04:25.127176 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.127182    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:04:25.127399 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.127190    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f962d8-4e78-401b-ba64-5b82a0de29c3-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:04:25.572444 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.572400    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:25.572973 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.572473    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:25.585986 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.585960    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:25.753184 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.753147    2575 generic.go:356] "Generic (PLEG): container finished" podID="9ca0c30b-11b4-464d-b28c-939e534e39f1" containerID="8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb" exitCode=0
Sep 16 19:04:25.753389 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.753198    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"
Sep 16 19:04:25.753389 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.753243    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l" event={"ID":"9ca0c30b-11b4-464d-b28c-939e534e39f1","Type":"ContainerDied","Data":"8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb"}
Sep 16 19:04:25.753389 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.753298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l" event={"ID":"9ca0c30b-11b4-464d-b28c-939e534e39f1","Type":"ContainerDied","Data":"4e518a21110ff6a2fed71faf54c8239d76cd3e27ec0612db967561543bbcec23"}
Sep 16 19:04:25.753389 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.753332    2575 scope.go:117] "RemoveContainer" containerID="8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb"
Sep 16 19:04:25.754821 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.754802    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"
Sep 16 19:04:25.754821 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.754805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs" event={"ID":"a1f962d8-4e78-401b-ba64-5b82a0de29c3","Type":"ContainerDied","Data":"7babf8da1b0bdc8a369da633082351116621794eb72157a0b5b8c55fa5302ba1"}
Sep 16 19:04:25.763452 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.763432    2575 scope.go:117] "RemoveContainer" containerID="8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb"
Sep 16 19:04:25.763806 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:04:25.763784    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb\": container with ID starting with 8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb not found: ID does not exist" containerID="8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb"
Sep 16 19:04:25.763886 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.763813    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb"} err="failed to get container status \"8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb\": rpc error: code = NotFound desc = could not find container \"8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb\": container with ID starting with 8317c899fa2b05b748c3d858a4761cde5a359ccfd211dddb34e4feee1cbcb8bb not found: ID does not exist"
Sep 16 19:04:25.763886 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.763830    2575 scope.go:117] "RemoveContainer" containerID="362dd983de0ef2e569a0d9ea03ded979da059f87c34723d383825179c4366a68"
Sep 16 19:04:25.771098 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.771076    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:04:25.784042 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.784013    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"]
Sep 16 19:04:25.786501 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.786482    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9d2450-5cf64c4cf4-pcp7l"]
Sep 16 19:04:25.801042 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.801010    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"]
Sep 16 19:04:25.810633 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:25.810599    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9d2450-5d4b646bbd-qssjs"]
Sep 16 19:04:26.623311 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:26.623220    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ca0c30b-11b4-464d-b28c-939e534e39f1" path="/var/lib/kubelet/pods/9ca0c30b-11b4-464d-b28c-939e534e39f1/volumes"
Sep 16 19:04:26.624169 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:26.624147    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1f962d8-4e78-401b-ba64-5b82a0de29c3" path="/var/lib/kubelet/pods/a1f962d8-4e78-401b-ba64-5b82a0de29c3/volumes"
Sep 16 19:04:26.727303 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:26.727274    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:04:31.723909 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:31.723873    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:04:44.773050 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.773005    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"]
Sep 16 19:04:44.773515 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.773500    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f962d8-4e78-401b-ba64-5b82a0de29c3" containerName="maas-api"
Sep 16 19:04:44.773566 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.773517    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f962d8-4e78-401b-ba64-5b82a0de29c3" containerName="maas-api"
Sep 16 19:04:44.773566 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.773541    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca0c30b-11b4-464d-b28c-939e534e39f1" containerName="payload-pre-processing"
Sep 16 19:04:44.773566 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.773547    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca0c30b-11b4-464d-b28c-939e534e39f1" containerName="payload-pre-processing"
Sep 16 19:04:44.773673 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.773619    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1f962d8-4e78-401b-ba64-5b82a0de29c3" containerName="maas-api"
Sep 16 19:04:44.773673 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.773631    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca0c30b-11b4-464d-b28c-939e534e39f1" containerName="payload-pre-processing"
Sep 16 19:04:44.776854 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.776831    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:44.782041 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.782016    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-58bf51-serving-cert\""
Sep 16 19:04:44.802311 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.802285    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"]
Sep 16 19:04:44.930153 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.930119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wtm\" (UniqueName: \"kubernetes.io/projected/e9598949-c34c-4b41-9ff8-0bec5d90df35-kube-api-access-52wtm\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:44.930329 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.930203    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:44.930329 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.930237    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:44.930329 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:44.930253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9598949-c34c-4b41-9ff8-0bec5d90df35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.030919 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.030840    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wtm\" (UniqueName: \"kubernetes.io/projected/e9598949-c34c-4b41-9ff8-0bec5d90df35-kube-api-access-52wtm\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.031078 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.030920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.031078 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.030947    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.031078 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.030963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9598949-c34c-4b41-9ff8-0bec5d90df35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.031637 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.031610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9598949-c34c-4b41-9ff8-0bec5d90df35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.033568 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.033544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.033683 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.033582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.041601 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.041573    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wtm\" (UniqueName: \"kubernetes.io/projected/e9598949-c34c-4b41-9ff8-0bec5d90df35-kube-api-access-52wtm\") pod \"maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.088160 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.088134    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.366084 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.366056    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"]
Sep 16 19:04:45.368362 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:04:45.368334    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9598949_c34c_4b41_9ff8_0bec5d90df35.slice/crio-00b04fb57da427d5c1f3e84f31d5423e735effd1df15981cda34d92eb16802e3 WatchSource:0}: Error finding container 00b04fb57da427d5c1f3e84f31d5423e735effd1df15981cda34d92eb16802e3: Status 404 returned error can't find the container with id 00b04fb57da427d5c1f3e84f31d5423e735effd1df15981cda34d92eb16802e3
Sep 16 19:04:45.548399 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.548364    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"]
Sep 16 19:04:45.553363 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.553343    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:04:45.557458 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.557434    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-58bf51-dockercfg-4q99k\""
Sep 16 19:04:45.557896 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.557869    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-58bf51\""
Sep 16 19:04:45.580290 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.580254    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"]
Sep 16 19:04:45.737741 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.737607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72425a16-256c-4a76-bf68-33032cbc96a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf\" (UID: \"72425a16-256c-4a76-bf68-33032cbc96a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:04:45.737933 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.737795    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkv6v\" (UniqueName: \"kubernetes.io/projected/72425a16-256c-4a76-bf68-33032cbc96a8-kube-api-access-mkv6v\") pod \"payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf\" (UID: \"72425a16-256c-4a76-bf68-33032cbc96a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:04:45.839001 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.838921    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkv6v\" (UniqueName: \"kubernetes.io/projected/72425a16-256c-4a76-bf68-33032cbc96a8-kube-api-access-mkv6v\") pod \"payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf\" (UID: \"72425a16-256c-4a76-bf68-33032cbc96a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:04:45.839430 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.839019    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72425a16-256c-4a76-bf68-33032cbc96a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf\" (UID: \"72425a16-256c-4a76-bf68-33032cbc96a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:04:45.839688 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.839664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72425a16-256c-4a76-bf68-33032cbc96a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf\" (UID: \"72425a16-256c-4a76-bf68-33032cbc96a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:04:45.848483 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.848460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkv6v\" (UniqueName: \"kubernetes.io/projected/72425a16-256c-4a76-bf68-33032cbc96a8-kube-api-access-mkv6v\") pod \"payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf\" (UID: \"72425a16-256c-4a76-bf68-33032cbc96a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:04:45.864839 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.864814    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:04:45.878024 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.877948    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk" event={"ID":"e9598949-c34c-4b41-9ff8-0bec5d90df35","Type":"ContainerStarted","Data":"e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862"}
Sep 16 19:04:45.878162 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.878063    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:45.878162 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.878106    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk" event={"ID":"e9598949-c34c-4b41-9ff8-0bec5d90df35","Type":"ContainerStarted","Data":"00b04fb57da427d5c1f3e84f31d5423e735effd1df15981cda34d92eb16802e3"}
Sep 16 19:04:45.897880 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:45.897726    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk" podStartSLOduration=1.497178017 podStartE2EDuration="1.897688505s" totalImagesPullingTime="400.510488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:45.896999832 +0000 UTC m=+1513.713765437" watchObservedRunningTime="2026-09-16 19:04:45.897688505 +0000 UTC m=+1513.714454095"
Sep 16 19:04:46.152590 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:46.152560    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"]
Sep 16 19:04:46.154544 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:04:46.154515    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72425a16_256c_4a76_bf68_33032cbc96a8.slice/crio-122db46634159b02bc135e541da74047aeb8db960ba9de6dc06a356a447cca87 WatchSource:0}: Error finding container 122db46634159b02bc135e541da74047aeb8db960ba9de6dc06a356a447cca87: Status 404 returned error can't find the container with id 122db46634159b02bc135e541da74047aeb8db960ba9de6dc06a356a447cca87
Sep 16 19:04:46.885073 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:46.885018    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf" event={"ID":"72425a16-256c-4a76-bf68-33032cbc96a8","Type":"ContainerStarted","Data":"7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2"}
Sep 16 19:04:46.885073 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:46.885076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf" event={"ID":"72425a16-256c-4a76-bf68-33032cbc96a8","Type":"ContainerStarted","Data":"122db46634159b02bc135e541da74047aeb8db960ba9de6dc06a356a447cca87"}
Sep 16 19:04:46.885586 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:46.885187    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:04:46.907089 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:46.907013    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf" podStartSLOduration=1.9069961979999999 podStartE2EDuration="1.906996198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:46.905701004 +0000 UTC m=+1514.722466589" watchObservedRunningTime="2026-09-16 19:04:46.906996198 +0000 UTC m=+1514.723761782"
Sep 16 19:04:51.890780 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:51.890748    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:04:57.894952 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:04:57.894879    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:05:06.728927 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:06.728887    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:05:20.513783 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.513738    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"]
Sep 16 19:05:20.514227 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.514007    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42" podUID="d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" containerName="maas-api" containerID="cri-o://00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa" gracePeriod=30
Sep 16 19:05:20.781555 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.781486    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 19:05:20.902521 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.902488    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-openshift-service-ca\") pod \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") "
Sep 16 19:05:20.902696 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.902554    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-maas-api-tls\") pod \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") "
Sep 16 19:05:20.902696 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.902674    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-metrics-certs\") pod \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") "
Sep 16 19:05:20.902864 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.902768    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89qkf\" (UniqueName: \"kubernetes.io/projected/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-kube-api-access-89qkf\") pod \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\" (UID: \"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af\") "
Sep 16 19:05:20.902971 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.902945    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" (UID: "d6afb5c7-ce99-4f4a-a296-ac1cf372d7af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:20.903109 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.903091    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:20.904761 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.904737    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" (UID: "d6afb5c7-ce99-4f4a-a296-ac1cf372d7af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:20.904858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.904801    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-kube-api-access-89qkf" (OuterVolumeSpecName: "kube-api-access-89qkf") pod "d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" (UID: "d6afb5c7-ce99-4f4a-a296-ac1cf372d7af"). InnerVolumeSpecName "kube-api-access-89qkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:20.904858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:20.904847    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" (UID: "d6afb5c7-ce99-4f4a-a296-ac1cf372d7af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:21.004141 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.004106    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:21.004141 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.004135    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qkf\" (UniqueName: \"kubernetes.io/projected/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-kube-api-access-89qkf\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:21.004141 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.004145    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:21.087615 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.087582    2575 generic.go:356] "Generic (PLEG): container finished" podID="d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" containerID="00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa" exitCode=0
Sep 16 19:05:21.087807 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.087629    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"
Sep 16 19:05:21.087807 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.087670    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42" event={"ID":"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af","Type":"ContainerDied","Data":"00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa"}
Sep 16 19:05:21.087807 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.087749    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42" event={"ID":"d6afb5c7-ce99-4f4a-a296-ac1cf372d7af","Type":"ContainerDied","Data":"ebe281848f08d3f6ed7688084a8c0beb04b493fb85782a9fa9edb3b3f4faf1ba"}
Sep 16 19:05:21.087807 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.087774    2575 scope.go:117] "RemoveContainer" containerID="00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa"
Sep 16 19:05:21.097665 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.097645    2575 scope.go:117] "RemoveContainer" containerID="00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa"
Sep 16 19:05:21.097974 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:05:21.097952    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa\": container with ID starting with 00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa not found: ID does not exist" containerID="00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa"
Sep 16 19:05:21.098053 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.097983    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa"} err="failed to get container status \"00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa\": rpc error: code = NotFound desc = could not find container \"00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa\": container with ID starting with 00cbda6afac90fe695351ff059100c3ba16c52dc1353d1bb1f2042d75b14dcaa not found: ID does not exist"
Sep 16 19:05:21.137000 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.136968    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"]
Sep 16 19:05:21.142780 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:21.142748    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec32e7-7fdcb9b64b-wkk42"]
Sep 16 19:05:22.623858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:22.623824    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" path="/var/lib/kubelet/pods/d6afb5c7-ce99-4f4a-a296-ac1cf372d7af/volumes"
Sep 16 19:05:27.714600 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.714567    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"]
Sep 16 19:05:27.715094 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.715071    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" containerName="maas-api"
Sep 16 19:05:27.715094 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.715091    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" containerName="maas-api"
Sep 16 19:05:27.715277 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.715238    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6afb5c7-ce99-4f4a-a296-ac1cf372d7af" containerName="maas-api"
Sep 16 19:05:27.720098 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.720076    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.722971 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.722950    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d929c607-serving-cert\""
Sep 16 19:05:27.729037 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.728701    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"]
Sep 16 19:05:27.779607 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.779584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g65hw\" (UniqueName: \"kubernetes.io/projected/13ca0c09-5f94-4a43-8374-fff629695027-kube-api-access-g65hw\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.779754 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.779640    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ca0c09-5f94-4a43-8374-fff629695027-openshift-service-ca\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.779754 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.779727    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-metrics-certs\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.779839 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.779815    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-maas-api-tls\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.881011 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.880982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ca0c09-5f94-4a43-8374-fff629695027-openshift-service-ca\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.881123 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.881031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-metrics-certs\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.881123 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.881088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-maas-api-tls\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.881200 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.881123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g65hw\" (UniqueName: \"kubernetes.io/projected/13ca0c09-5f94-4a43-8374-fff629695027-kube-api-access-g65hw\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.881252 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:05:27.881222    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-serving-cert: secret "maas-api-e2e-mt-d929c607-serving-cert" not found
Sep 16 19:05:27.881319 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:05:27.881307    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-maas-api-tls podName:13ca0c09-5f94-4a43-8374-fff629695027 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:28.381286534 +0000 UTC m=+1556.198052098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-maas-api-tls") pod "maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v" (UID: "13ca0c09-5f94-4a43-8374-fff629695027") : secret "maas-api-e2e-mt-d929c607-serving-cert" not found
Sep 16 19:05:27.881629 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.881610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ca0c09-5f94-4a43-8374-fff629695027-openshift-service-ca\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.883535 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.883513    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-metrics-certs\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:27.891053 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:27.891029    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g65hw\" (UniqueName: \"kubernetes.io/projected/13ca0c09-5f94-4a43-8374-fff629695027-kube-api-access-g65hw\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:28.387216 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:28.387180    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-maas-api-tls\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:28.389671 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:28.389645    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-maas-api-tls\") pod \"maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:28.633119 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:28.633082    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:28.921082 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:28.921054    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"]
Sep 16 19:05:28.922831 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:05:28.922799    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ca0c09_5f94_4a43_8374_fff629695027.slice/crio-4f56cde4cbf55776a8ca6654a665fc9b4ccb034c2cea70b094bc124f03a6f330 WatchSource:0}: Error finding container 4f56cde4cbf55776a8ca6654a665fc9b4ccb034c2cea70b094bc124f03a6f330: Status 404 returned error can't find the container with id 4f56cde4cbf55776a8ca6654a665fc9b4ccb034c2cea70b094bc124f03a6f330
Sep 16 19:05:29.135559 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:29.135520    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v" event={"ID":"13ca0c09-5f94-4a43-8374-fff629695027","Type":"ContainerStarted","Data":"4f56cde4cbf55776a8ca6654a665fc9b4ccb034c2cea70b094bc124f03a6f330"}
Sep 16 19:05:30.146318 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:30.146264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v" event={"ID":"13ca0c09-5f94-4a43-8374-fff629695027","Type":"ContainerStarted","Data":"db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3"}
Sep 16 19:05:30.146758 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:30.146394    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:30.178934 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:30.178884    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v" podStartSLOduration=2.717373666 podStartE2EDuration="3.178870174s" totalImagesPullingTime="461.496508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:30.178454095 +0000 UTC m=+1557.995219681" watchObservedRunningTime="2026-09-16 19:05:30.178870174 +0000 UTC m=+1557.995635760"
Sep 16 19:05:35.534110 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:35.534072    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:05:36.157039 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:36.157006    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:05:41.364671 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.364626    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"]
Sep 16 19:05:41.365269 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.365233    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c" podUID="ea1c1592-509d-47c9-ad1b-607c73d3ea59" containerName="maas-api" containerID="cri-o://4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155" gracePeriod=30
Sep 16 19:05:41.437851 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.437809    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"]
Sep 16 19:05:41.438129 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.438102    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6" podUID="f11b2aeb-ecde-47b6-999b-e52aaf7725b7" containerName="payload-processing" containerID="cri-o://fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298" gracePeriod=30
Sep 16 19:05:41.447322 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.447281    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"]
Sep 16 19:05:41.447555 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.447525    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98" podUID="ea2108f5-f17c-4377-9730-2f990ef6b4b2" containerName="payload-pre-processing" containerID="cri-o://fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7" gracePeriod=30
Sep 16 19:05:41.736511 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.736477    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 19:05:41.762306 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.762269    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 19:05:41.772198 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.772163    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 19:05:41.829686 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.829644    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-metrics-certs\") pod \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") "
Sep 16 19:05:41.829900 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.829721    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-plugins-config-volume\") pod \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\" (UID: \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\") "
Sep 16 19:05:41.829900 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.829757    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2108f5-f17c-4377-9730-2f990ef6b4b2-plugins-config-volume\") pod \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\" (UID: \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\") "
Sep 16 19:05:41.829900 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.829783    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fwqv\" (UniqueName: \"kubernetes.io/projected/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-kube-api-access-8fwqv\") pod \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\" (UID: \"f11b2aeb-ecde-47b6-999b-e52aaf7725b7\") "
Sep 16 19:05:41.830095 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.829900    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z456g\" (UniqueName: \"kubernetes.io/projected/ea2108f5-f17c-4377-9730-2f990ef6b4b2-kube-api-access-z456g\") pod \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\" (UID: \"ea2108f5-f17c-4377-9730-2f990ef6b4b2\") "
Sep 16 19:05:41.830095 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.829962    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrc2g\" (UniqueName: \"kubernetes.io/projected/ea1c1592-509d-47c9-ad1b-607c73d3ea59-kube-api-access-xrc2g\") pod \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") "
Sep 16 19:05:41.830095 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.830009    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-maas-api-tls\") pod \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") "
Sep 16 19:05:41.830095 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.830039    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c1592-509d-47c9-ad1b-607c73d3ea59-openshift-service-ca\") pod \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\" (UID: \"ea1c1592-509d-47c9-ad1b-607c73d3ea59\") "
Sep 16 19:05:41.830312 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.830183    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2108f5-f17c-4377-9730-2f990ef6b4b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea2108f5-f17c-4377-9730-2f990ef6b4b2" (UID: "ea2108f5-f17c-4377-9730-2f990ef6b4b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:41.830506 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.830484    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1c1592-509d-47c9-ad1b-607c73d3ea59-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea1c1592-509d-47c9-ad1b-607c73d3ea59" (UID: "ea1c1592-509d-47c9-ad1b-607c73d3ea59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:41.830585 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.830538    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f11b2aeb-ecde-47b6-999b-e52aaf7725b7" (UID: "f11b2aeb-ecde-47b6-999b-e52aaf7725b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:41.830645 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.830597    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2108f5-f17c-4377-9730-2f990ef6b4b2-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:41.830645 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.830612    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1c1592-509d-47c9-ad1b-607c73d3ea59-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:41.832785 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.832740    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2108f5-f17c-4377-9730-2f990ef6b4b2-kube-api-access-z456g" (OuterVolumeSpecName: "kube-api-access-z456g") pod "ea2108f5-f17c-4377-9730-2f990ef6b4b2" (UID: "ea2108f5-f17c-4377-9730-2f990ef6b4b2"). InnerVolumeSpecName "kube-api-access-z456g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:41.832785 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.832745    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-kube-api-access-8fwqv" (OuterVolumeSpecName: "kube-api-access-8fwqv") pod "f11b2aeb-ecde-47b6-999b-e52aaf7725b7" (UID: "f11b2aeb-ecde-47b6-999b-e52aaf7725b7"). InnerVolumeSpecName "kube-api-access-8fwqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:41.833052 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.833032    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea1c1592-509d-47c9-ad1b-607c73d3ea59" (UID: "ea1c1592-509d-47c9-ad1b-607c73d3ea59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:41.833154 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.833126    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1c1592-509d-47c9-ad1b-607c73d3ea59-kube-api-access-xrc2g" (OuterVolumeSpecName: "kube-api-access-xrc2g") pod "ea1c1592-509d-47c9-ad1b-607c73d3ea59" (UID: "ea1c1592-509d-47c9-ad1b-607c73d3ea59"). InnerVolumeSpecName "kube-api-access-xrc2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:41.833308 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.833290    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea1c1592-509d-47c9-ad1b-607c73d3ea59" (UID: "ea1c1592-509d-47c9-ad1b-607c73d3ea59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:41.931359 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.931272    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z456g\" (UniqueName: \"kubernetes.io/projected/ea2108f5-f17c-4377-9730-2f990ef6b4b2-kube-api-access-z456g\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:41.931359 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.931302    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrc2g\" (UniqueName: \"kubernetes.io/projected/ea1c1592-509d-47c9-ad1b-607c73d3ea59-kube-api-access-xrc2g\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:41.931359 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.931312    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:41.931359 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.931324    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea1c1592-509d-47c9-ad1b-607c73d3ea59-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:41.931359 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.931336    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:41.931359 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:41.931344    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fwqv\" (UniqueName: \"kubernetes.io/projected/f11b2aeb-ecde-47b6-999b-e52aaf7725b7-kube-api-access-8fwqv\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:05:42.218154 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.218063    2575 generic.go:356] "Generic (PLEG): container finished" podID="f11b2aeb-ecde-47b6-999b-e52aaf7725b7" containerID="fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298" exitCode=0
Sep 16 19:05:42.218154 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.218108    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"
Sep 16 19:05:42.218350 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.218146    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6" event={"ID":"f11b2aeb-ecde-47b6-999b-e52aaf7725b7","Type":"ContainerDied","Data":"fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298"}
Sep 16 19:05:42.218350 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.218214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6" event={"ID":"f11b2aeb-ecde-47b6-999b-e52aaf7725b7","Type":"ContainerDied","Data":"add78d46a3aa4f04ac02d950bf6b22793b4cb91fc07c739e2da98db2154d669e"}
Sep 16 19:05:42.218350 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.218249    2575 scope.go:117] "RemoveContainer" containerID="fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298"
Sep 16 19:05:42.219590 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.219570    2575 generic.go:356] "Generic (PLEG): container finished" podID="ea1c1592-509d-47c9-ad1b-607c73d3ea59" containerID="4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155" exitCode=0
Sep 16 19:05:42.219669 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.219608    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"
Sep 16 19:05:42.219669 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.219629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c" event={"ID":"ea1c1592-509d-47c9-ad1b-607c73d3ea59","Type":"ContainerDied","Data":"4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155"}
Sep 16 19:05:42.219809 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.219680    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c" event={"ID":"ea1c1592-509d-47c9-ad1b-607c73d3ea59","Type":"ContainerDied","Data":"7a6434495fd6fa1b25a6f966a5c435826d0403f55472218339b2ea6514f7b871"}
Sep 16 19:05:42.221218 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.221182    2575 generic.go:356] "Generic (PLEG): container finished" podID="ea2108f5-f17c-4377-9730-2f990ef6b4b2" containerID="fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7" exitCode=0
Sep 16 19:05:42.221324 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.221213    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98" event={"ID":"ea2108f5-f17c-4377-9730-2f990ef6b4b2","Type":"ContainerDied","Data":"fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7"}
Sep 16 19:05:42.221324 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.221234    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"
Sep 16 19:05:42.221324 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.221263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98" event={"ID":"ea2108f5-f17c-4377-9730-2f990ef6b4b2","Type":"ContainerDied","Data":"a111e5a392ff235287eb55954c85d6f4c934fa04def40ef4b3fcd142650271fa"}
Sep 16 19:05:42.229177 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.229157    2575 scope.go:117] "RemoveContainer" containerID="fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298"
Sep 16 19:05:42.229584 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:05:42.229553    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298\": container with ID starting with fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298 not found: ID does not exist" containerID="fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298"
Sep 16 19:05:42.229684 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.229596    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298"} err="failed to get container status \"fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298\": rpc error: code = NotFound desc = could not find container \"fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298\": container with ID starting with fd96680c14040f4ab906b7e08bf206374fa6d52fc27726daab78f607268d3298 not found: ID does not exist"
Sep 16 19:05:42.229684 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.229623    2575 scope.go:117] "RemoveContainer" containerID="4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155"
Sep 16 19:05:42.239088 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.239069    2575 scope.go:117] "RemoveContainer" containerID="4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155"
Sep 16 19:05:42.239387 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:05:42.239370    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155\": container with ID starting with 4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155 not found: ID does not exist" containerID="4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155"
Sep 16 19:05:42.239440 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.239397    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155"} err="failed to get container status \"4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155\": rpc error: code = NotFound desc = could not find container \"4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155\": container with ID starting with 4b2ded7cc0fea4bc37c13ef30494cb901399d9ec8c4d05df8fddacc638667155 not found: ID does not exist"
Sep 16 19:05:42.239440 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.239414    2575 scope.go:117] "RemoveContainer" containerID="fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7"
Sep 16 19:05:42.248685 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.248662    2575 scope.go:117] "RemoveContainer" containerID="fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7"
Sep 16 19:05:42.249070 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:05:42.249051    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7\": container with ID starting with fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7 not found: ID does not exist" containerID="fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7"
Sep 16 19:05:42.249136 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.249078    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7"} err="failed to get container status \"fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7\": rpc error: code = NotFound desc = could not find container \"fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7\": container with ID starting with fc0b9ecf5fcf427189f77f82529b9f693f7aad98aa07782f43610935810864a7 not found: ID does not exist"
Sep 16 19:05:42.249865 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.249844    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"]
Sep 16 19:05:42.255555 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.255526    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a45a9-79b6765f75-rmqw6"]
Sep 16 19:05:42.268626 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.268581    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"]
Sep 16 19:05:42.272629 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.272599    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a45a9-6f94b54654-f7f98"]
Sep 16 19:05:42.285575 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.285544    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"]
Sep 16 19:05:42.291101 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.291073    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a45a9-5859bf59fd-6cf8c"]
Sep 16 19:05:42.622805 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.622764    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea1c1592-509d-47c9-ad1b-607c73d3ea59" path="/var/lib/kubelet/pods/ea1c1592-509d-47c9-ad1b-607c73d3ea59/volumes"
Sep 16 19:05:42.623188 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.623148    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2108f5-f17c-4377-9730-2f990ef6b4b2" path="/var/lib/kubelet/pods/ea2108f5-f17c-4377-9730-2f990ef6b4b2/volumes"
Sep 16 19:05:42.623492 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:42.623478    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f11b2aeb-ecde-47b6-999b-e52aaf7725b7" path="/var/lib/kubelet/pods/f11b2aeb-ecde-47b6-999b-e52aaf7725b7/volumes"
Sep 16 19:05:56.375062 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.375022    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"]
Sep 16 19:05:56.375835 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.375814    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2108f5-f17c-4377-9730-2f990ef6b4b2" containerName="payload-pre-processing"
Sep 16 19:05:56.375835 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.375837    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2108f5-f17c-4377-9730-2f990ef6b4b2" containerName="payload-pre-processing"
Sep 16 19:05:56.375994 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.375856    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f11b2aeb-ecde-47b6-999b-e52aaf7725b7" containerName="payload-processing"
Sep 16 19:05:56.375994 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.375865    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f11b2aeb-ecde-47b6-999b-e52aaf7725b7" containerName="payload-processing"
Sep 16 19:05:56.375994 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.375880    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1c1592-509d-47c9-ad1b-607c73d3ea59" containerName="maas-api"
Sep 16 19:05:56.375994 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.375889    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1c1592-509d-47c9-ad1b-607c73d3ea59" containerName="maas-api"
Sep 16 19:05:56.375994 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.375994    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea1c1592-509d-47c9-ad1b-607c73d3ea59" containerName="maas-api"
Sep 16 19:05:56.376234 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.376008    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2108f5-f17c-4377-9730-2f990ef6b4b2" containerName="payload-pre-processing"
Sep 16 19:05:56.376234 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.376027    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f11b2aeb-ecde-47b6-999b-e52aaf7725b7" containerName="payload-processing"
Sep 16 19:05:56.381059 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.381035    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.384565 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.384539    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c190a0\"/\"default-dockercfg-xj65n\""
Sep 16 19:05:56.384695 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.384540    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c190a0\"/\"openshift-service-ca.crt\""
Sep 16 19:05:56.384695 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.384547    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c190a0\"/\"noten-model-c190a0-kserve-self-signed-certs\""
Sep 16 19:05:56.385483 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.385038    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c190a0\"/\"kube-root-ca.crt\""
Sep 16 19:05:56.387541 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.387516    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"]
Sep 16 19:05:56.483909 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.483856    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-model-cache\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.484092 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.483944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgc2x\" (UniqueName: \"kubernetes.io/projected/cbb38aff-d0e4-425c-aa35-66313ff2002f-kube-api-access-fgc2x\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.484092 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.483989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-tmp-dir\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.484092 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.484025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-dshm\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.484239 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.484098    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-home\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.484239 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.484160    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb38aff-d0e4-425c-aa35-66313ff2002f-tls-certs\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.585788 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.585749    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-dshm\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.585788 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.585792    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-home\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.586049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.585841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb38aff-d0e4-425c-aa35-66313ff2002f-tls-certs\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.586049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.585937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-model-cache\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.586049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.585968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgc2x\" (UniqueName: \"kubernetes.io/projected/cbb38aff-d0e4-425c-aa35-66313ff2002f-kube-api-access-fgc2x\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.586049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.586006    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-tmp-dir\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.586306 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.586284    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-home\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.586306 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.586298    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-tmp-dir\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.586548 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.586522    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-model-cache\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.588488 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.588458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb38aff-d0e4-425c-aa35-66313ff2002f-tls-certs\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.588615 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.588584    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-dshm\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.597861 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.597829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgc2x\" (UniqueName: \"kubernetes.io/projected/cbb38aff-d0e4-425c-aa35-66313ff2002f-kube-api-access-fgc2x\") pod \"noten-model-c190a0-kserve-7cc87c57cf-k9mrq\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") " pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.692600 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.692517    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:05:56.978510 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:56.978433    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"]
Sep 16 19:05:56.983540 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:05:56.983502    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbb38aff_d0e4_425c_aa35_66313ff2002f.slice/crio-f7bc728f0c71eb314cb5e8ae27de6b3127d3a9f0c22c29ac14697d9e2e93f55a WatchSource:0}: Error finding container f7bc728f0c71eb314cb5e8ae27de6b3127d3a9f0c22c29ac14697d9e2e93f55a: Status 404 returned error can't find the container with id f7bc728f0c71eb314cb5e8ae27de6b3127d3a9f0c22c29ac14697d9e2e93f55a
Sep 16 19:05:57.319173 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:57.319074    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq" event={"ID":"cbb38aff-d0e4-425c-aa35-66313ff2002f","Type":"ContainerStarted","Data":"3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c"}
Sep 16 19:05:57.319173 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:57.319125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq" event={"ID":"cbb38aff-d0e4-425c-aa35-66313ff2002f","Type":"ContainerStarted","Data":"f7bc728f0c71eb314cb5e8ae27de6b3127d3a9f0c22c29ac14697d9e2e93f55a"}
Sep 16 19:05:57.341792 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:57.341743    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq" podStartSLOduration=1.341726116 podStartE2EDuration="1.341726116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:57.339979535 +0000 UTC m=+1585.156745134" watchObservedRunningTime="2026-09-16 19:05:57.341726116 +0000 UTC m=+1585.158491703"
Sep 16 19:05:59.453324 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.453293    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"]
Sep 16 19:05:59.457040 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.457023    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.461757 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.461738    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-ca3f9a-serving-cert\""
Sep 16 19:05:59.477760 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.477734    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"]
Sep 16 19:05:59.514313 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.514284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74l4h\" (UniqueName: \"kubernetes.io/projected/d3f14aab-a37f-4766-8704-67abd2df8313-kube-api-access-74l4h\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.514472 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.514318    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.514472 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.514340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.514472 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.514435    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f14aab-a37f-4766-8704-67abd2df8313-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.615448 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.615413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74l4h\" (UniqueName: \"kubernetes.io/projected/d3f14aab-a37f-4766-8704-67abd2df8313-kube-api-access-74l4h\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.615448 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.615453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.615670 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.615475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.615670 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.615517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f14aab-a37f-4766-8704-67abd2df8313-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.616222 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.616195    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f14aab-a37f-4766-8704-67abd2df8313-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.617931 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.617898    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.618045 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.618000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.630906 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.630879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74l4h\" (UniqueName: \"kubernetes.io/projected/d3f14aab-a37f-4766-8704-67abd2df8313-kube-api-access-74l4h\") pod \"maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:05:59.767379 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:05:59.767302    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:06:00.046002 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.045976    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"]
Sep 16 19:06:00.047257 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:06:00.047224    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f14aab_a37f_4766_8704_67abd2df8313.slice/crio-172d97d734add090543e304f2c514ae0121db0de0c6eb3750f52d94705bce207 WatchSource:0}: Error finding container 172d97d734add090543e304f2c514ae0121db0de0c6eb3750f52d94705bce207: Status 404 returned error can't find the container with id 172d97d734add090543e304f2c514ae0121db0de0c6eb3750f52d94705bce207
Sep 16 19:06:00.268248 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.268217    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"]
Sep 16 19:06:00.273374 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.273346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:00.276941 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.276917    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ca3f9a-dockercfg-9c6t7\""
Sep 16 19:06:00.277155 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.277133    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ca3f9a\""
Sep 16 19:06:00.285685 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.285654    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"]
Sep 16 19:06:00.323054 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.322953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77nj9\" (UniqueName: \"kubernetes.io/projected/bab2f95b-84c8-432a-842b-60889a41cb14-kube-api-access-77nj9\") pod \"payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r\" (UID: \"bab2f95b-84c8-432a-842b-60889a41cb14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:00.323214 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.323139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab2f95b-84c8-432a-842b-60889a41cb14-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r\" (UID: \"bab2f95b-84c8-432a-842b-60889a41cb14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:00.338191 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.338157    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2" event={"ID":"d3f14aab-a37f-4766-8704-67abd2df8313","Type":"ContainerStarted","Data":"172d97d734add090543e304f2c514ae0121db0de0c6eb3750f52d94705bce207"}
Sep 16 19:06:00.424428 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.424359    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab2f95b-84c8-432a-842b-60889a41cb14-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r\" (UID: \"bab2f95b-84c8-432a-842b-60889a41cb14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:00.424616 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.424498    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77nj9\" (UniqueName: \"kubernetes.io/projected/bab2f95b-84c8-432a-842b-60889a41cb14-kube-api-access-77nj9\") pod \"payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r\" (UID: \"bab2f95b-84c8-432a-842b-60889a41cb14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:00.425152 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.425127    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab2f95b-84c8-432a-842b-60889a41cb14-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r\" (UID: \"bab2f95b-84c8-432a-842b-60889a41cb14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:00.436114 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.436081    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77nj9\" (UniqueName: \"kubernetes.io/projected/bab2f95b-84c8-432a-842b-60889a41cb14-kube-api-access-77nj9\") pod \"payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r\" (UID: \"bab2f95b-84c8-432a-842b-60889a41cb14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:00.585374 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.585276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:00.883246 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:00.883216    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"]
Sep 16 19:06:00.885518 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:06:00.885487    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab2f95b_84c8_432a_842b_60889a41cb14.slice/crio-0e6792d260e215af9cc4d81418e99d46768febc3fb95ee735a5dc50622764fba WatchSource:0}: Error finding container 0e6792d260e215af9cc4d81418e99d46768febc3fb95ee735a5dc50622764fba: Status 404 returned error can't find the container with id 0e6792d260e215af9cc4d81418e99d46768febc3fb95ee735a5dc50622764fba
Sep 16 19:06:01.345991 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:01.345952    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r" event={"ID":"bab2f95b-84c8-432a-842b-60889a41cb14","Type":"ContainerStarted","Data":"827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac"}
Sep 16 19:06:01.346213 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:01.346018    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r" event={"ID":"bab2f95b-84c8-432a-842b-60889a41cb14","Type":"ContainerStarted","Data":"0e6792d260e215af9cc4d81418e99d46768febc3fb95ee735a5dc50622764fba"}
Sep 16 19:06:01.346213 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:01.346135    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:01.347855 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:01.347825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2" event={"ID":"d3f14aab-a37f-4766-8704-67abd2df8313","Type":"ContainerStarted","Data":"ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c"}
Sep 16 19:06:01.348021 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:01.347945    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:06:01.367262 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:01.367192    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r" podStartSLOduration=1.367169551 podStartE2EDuration="1.367169551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:01.366533609 +0000 UTC m=+1589.183299197" watchObservedRunningTime="2026-09-16 19:06:01.367169551 +0000 UTC m=+1589.183935138"
Sep 16 19:06:06.692757 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:06.692721    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:06:06.693161 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:06.692787    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:06:06.705914 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:06.705883    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:06:06.726924 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:06.726874    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2" podStartSLOduration=7.328726003 podStartE2EDuration="7.726857213s" totalImagesPullingTime="398.13121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:01.390120214 +0000 UTC m=+1589.206885829" watchObservedRunningTime="2026-09-16 19:06:06.726857213 +0000 UTC m=+1594.543622799"
Sep 16 19:06:07.358773 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:07.358744    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:06:07.393765 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:07.393728    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:06:12.355155 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:12.355122    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:06:18.800121 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:18.800085    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:06:33.997493 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:33.997457    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"]
Sep 16 19:06:36.769520 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.769488    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"]
Sep 16 19:06:36.773334 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.773313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.779833 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.779811    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-2dc9fc-serving-cert\""
Sep 16 19:06:36.793249 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.793222    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"]
Sep 16 19:06:36.803649 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.803618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.803786 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.803677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f63f930-c84d-4c94-b814-9443d2e4d07e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.803838 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.803790    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4v5\" (UniqueName: \"kubernetes.io/projected/8f63f930-c84d-4c94-b814-9443d2e4d07e-kube-api-access-tq4v5\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.803900 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.803882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.904843 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.904809    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f63f930-c84d-4c94-b814-9443d2e4d07e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.905039 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.904891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4v5\" (UniqueName: \"kubernetes.io/projected/8f63f930-c84d-4c94-b814-9443d2e4d07e-kube-api-access-tq4v5\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.905039 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.904982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.905150 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.905079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.905234 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:06:36.905218    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-serving-cert: secret "maas-api-e2e-shared-b-w3-2dc9fc-serving-cert" not found
Sep 16 19:06:36.905298 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:06:36.905286    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-maas-api-tls podName:8f63f930-c84d-4c94-b814-9443d2e4d07e nodeName:}" failed. No retries permitted until 2026-09-16 19:06:37.405266133 +0000 UTC m=+1625.222031698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-maas-api-tls") pod "maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9" (UID: "8f63f930-c84d-4c94-b814-9443d2e4d07e") : secret "maas-api-e2e-shared-b-w3-2dc9fc-serving-cert" not found
Sep 16 19:06:36.905651 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.905628    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f63f930-c84d-4c94-b814-9443d2e4d07e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.907772 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.907745    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:36.928594 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:36.928565    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4v5\" (UniqueName: \"kubernetes.io/projected/8f63f930-c84d-4c94-b814-9443d2e4d07e-kube-api-access-tq4v5\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:37.410563 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:37.410519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:37.413028 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:37.412999    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:37.684806 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:37.684724    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:37.908811 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:37.908779    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"]
Sep 16 19:06:37.912817 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:37.912793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:37.917133 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:37.917109    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2dc9fc\""
Sep 16 19:06:37.917243 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:37.917112    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2dc9fc-dockercfg-5jptv\""
Sep 16 19:06:37.937994 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:37.937937    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"]
Sep 16 19:06:37.993159 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:37.993138    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"]
Sep 16 19:06:37.995504 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:06:37.995480    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f63f930_c84d_4c94_b814_9443d2e4d07e.slice/crio-1f89b0bd40829790a3ee05e524fb4ce42324ecc101fd5bc842824f866b754251 WatchSource:0}: Error finding container 1f89b0bd40829790a3ee05e524fb4ce42324ecc101fd5bc842824f866b754251: Status 404 returned error can't find the container with id 1f89b0bd40829790a3ee05e524fb4ce42324ecc101fd5bc842824f866b754251
Sep 16 19:06:38.018671 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.018638    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e272518-f9af-4a70-98f9-3d7d474edd13-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd\" (UID: \"2e272518-f9af-4a70-98f9-3d7d474edd13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:38.018801 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.018729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdrm\" (UniqueName: \"kubernetes.io/projected/2e272518-f9af-4a70-98f9-3d7d474edd13-kube-api-access-cqdrm\") pod \"payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd\" (UID: \"2e272518-f9af-4a70-98f9-3d7d474edd13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:38.120089 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.120052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e272518-f9af-4a70-98f9-3d7d474edd13-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd\" (UID: \"2e272518-f9af-4a70-98f9-3d7d474edd13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:38.120229 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.120136    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdrm\" (UniqueName: \"kubernetes.io/projected/2e272518-f9af-4a70-98f9-3d7d474edd13-kube-api-access-cqdrm\") pod \"payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd\" (UID: \"2e272518-f9af-4a70-98f9-3d7d474edd13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:38.120759 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.120739    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e272518-f9af-4a70-98f9-3d7d474edd13-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd\" (UID: \"2e272518-f9af-4a70-98f9-3d7d474edd13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:38.133573 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.133538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdrm\" (UniqueName: \"kubernetes.io/projected/2e272518-f9af-4a70-98f9-3d7d474edd13-kube-api-access-cqdrm\") pod \"payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd\" (UID: \"2e272518-f9af-4a70-98f9-3d7d474edd13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:38.223761 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.223673    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:38.504968 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.504945    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"]
Sep 16 19:06:38.521019 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:06:38.520985    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e272518_f9af_4a70_98f9_3d7d474edd13.slice/crio-471c6884e5258992888526b41b463b14278bd5731b8321f3b4c74b2dc3619202 WatchSource:0}: Error finding container 471c6884e5258992888526b41b463b14278bd5731b8321f3b4c74b2dc3619202: Status 404 returned error can't find the container with id 471c6884e5258992888526b41b463b14278bd5731b8321f3b4c74b2dc3619202
Sep 16 19:06:38.561289 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.561259    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd" event={"ID":"2e272518-f9af-4a70-98f9-3d7d474edd13","Type":"ContainerStarted","Data":"471c6884e5258992888526b41b463b14278bd5731b8321f3b4c74b2dc3619202"}
Sep 16 19:06:38.563431 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.563402    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9" event={"ID":"8f63f930-c84d-4c94-b814-9443d2e4d07e","Type":"ContainerStarted","Data":"5882cefb90d62e45278a9534ac0fbb9c368461079ecb4c4dc138baa899e2cb6a"}
Sep 16 19:06:38.563563 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.563454    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9" event={"ID":"8f63f930-c84d-4c94-b814-9443d2e4d07e","Type":"ContainerStarted","Data":"1f89b0bd40829790a3ee05e524fb4ce42324ecc101fd5bc842824f866b754251"}
Sep 16 19:06:38.563563 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.563542    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:38.593446 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:38.593393    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9" podStartSLOduration=2.133502599 podStartE2EDuration="2.593374673s" totalImagesPullingTime="459.872074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:38.591085289 +0000 UTC m=+1626.407850872" watchObservedRunningTime="2026-09-16 19:06:38.593374673 +0000 UTC m=+1626.410140261"
Sep 16 19:06:39.570807 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:39.570762    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd" event={"ID":"2e272518-f9af-4a70-98f9-3d7d474edd13","Type":"ContainerStarted","Data":"3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162"}
Sep 16 19:06:39.571220 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:39.570841    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:39.609287 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:39.609233    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd" podStartSLOduration=2.609216967 podStartE2EDuration="2.609216967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:39.606243819 +0000 UTC m=+1627.423009406" watchObservedRunningTime="2026-09-16 19:06:39.609216967 +0000 UTC m=+1627.425982553"
Sep 16 19:06:44.576741 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:44.576690    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:06:49.016589 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.016543    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq" podUID="cbb38aff-d0e4-425c-aa35-66313ff2002f" containerName="main" containerID="cri-o://3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c" gracePeriod=45
Sep 16 19:06:49.274166 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.274100    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:06:49.334843 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.334817    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-tmp-dir\") pod \"cbb38aff-d0e4-425c-aa35-66313ff2002f\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") "
Sep 16 19:06:49.334986 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.334850    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb38aff-d0e4-425c-aa35-66313ff2002f-tls-certs\") pod \"cbb38aff-d0e4-425c-aa35-66313ff2002f\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") "
Sep 16 19:06:49.334986 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.334920    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-home\") pod \"cbb38aff-d0e4-425c-aa35-66313ff2002f\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") "
Sep 16 19:06:49.334986 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.334965    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgc2x\" (UniqueName: \"kubernetes.io/projected/cbb38aff-d0e4-425c-aa35-66313ff2002f-kube-api-access-fgc2x\") pod \"cbb38aff-d0e4-425c-aa35-66313ff2002f\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") "
Sep 16 19:06:49.335166 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.335055    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-model-cache\") pod \"cbb38aff-d0e4-425c-aa35-66313ff2002f\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") "
Sep 16 19:06:49.335166 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.335079    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-dshm\") pod \"cbb38aff-d0e4-425c-aa35-66313ff2002f\" (UID: \"cbb38aff-d0e4-425c-aa35-66313ff2002f\") "
Sep 16 19:06:49.335266 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.335068    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbb38aff-d0e4-425c-aa35-66313ff2002f" (UID: "cbb38aff-d0e4-425c-aa35-66313ff2002f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:49.335318 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.335136    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-home" (OuterVolumeSpecName: "home") pod "cbb38aff-d0e4-425c-aa35-66313ff2002f" (UID: "cbb38aff-d0e4-425c-aa35-66313ff2002f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:49.335318 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.335227    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbb38aff-d0e4-425c-aa35-66313ff2002f" (UID: "cbb38aff-d0e4-425c-aa35-66313ff2002f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:49.335491 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.335473    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-tmp-dir\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:06:49.335571 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.335494    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-home\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:06:49.335571 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.335507    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-model-cache\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:06:49.337168 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.337112    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb38aff-d0e4-425c-aa35-66313ff2002f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbb38aff-d0e4-425c-aa35-66313ff2002f" (UID: "cbb38aff-d0e4-425c-aa35-66313ff2002f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:49.337168 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.337154    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-dshm" (OuterVolumeSpecName: "dshm") pod "cbb38aff-d0e4-425c-aa35-66313ff2002f" (UID: "cbb38aff-d0e4-425c-aa35-66313ff2002f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:49.337500 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.337476    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb38aff-d0e4-425c-aa35-66313ff2002f-kube-api-access-fgc2x" (OuterVolumeSpecName: "kube-api-access-fgc2x") pod "cbb38aff-d0e4-425c-aa35-66313ff2002f" (UID: "cbb38aff-d0e4-425c-aa35-66313ff2002f"). InnerVolumeSpecName "kube-api-access-fgc2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:49.436787 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.436753    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb38aff-d0e4-425c-aa35-66313ff2002f-tls-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:06:49.436787 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.436784    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgc2x\" (UniqueName: \"kubernetes.io/projected/cbb38aff-d0e4-425c-aa35-66313ff2002f-kube-api-access-fgc2x\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:06:49.436937 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.436794    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb38aff-d0e4-425c-aa35-66313ff2002f-dshm\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:06:49.634621 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.634595    2575 generic.go:356] "Generic (PLEG): container finished" podID="cbb38aff-d0e4-425c-aa35-66313ff2002f" containerID="3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c" exitCode=0
Sep 16 19:06:49.634794 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.634634    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"
Sep 16 19:06:49.634794 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.634682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq" event={"ID":"cbb38aff-d0e4-425c-aa35-66313ff2002f","Type":"ContainerDied","Data":"3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c"}
Sep 16 19:06:49.634794 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.634733    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq" event={"ID":"cbb38aff-d0e4-425c-aa35-66313ff2002f","Type":"ContainerDied","Data":"f7bc728f0c71eb314cb5e8ae27de6b3127d3a9f0c22c29ac14697d9e2e93f55a"}
Sep 16 19:06:49.634794 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.634757    2575 scope.go:117] "RemoveContainer" containerID="3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c"
Sep 16 19:06:49.644236 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.644219    2575 scope.go:117] "RemoveContainer" containerID="3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c"
Sep 16 19:06:49.644493 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:06:49.644473    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c\": container with ID starting with 3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c not found: ID does not exist" containerID="3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c"
Sep 16 19:06:49.644550 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.644506    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c"} err="failed to get container status \"3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c\": rpc error: code = NotFound desc = could not find container \"3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c\": container with ID starting with 3954fd4d30fadf35588eccaa47b7e10eee22ff32542b56a77a449b46fd64512c not found: ID does not exist"
Sep 16 19:06:49.667837 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.667817    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"]
Sep 16 19:06:49.670027 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:49.670008    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c190a0/noten-model-c190a0-kserve-7cc87c57cf-k9mrq"]
Sep 16 19:06:50.582909 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:50.582875    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:06:50.623783 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:50.623747    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb38aff-d0e4-425c-aa35-66313ff2002f" path="/var/lib/kubelet/pods/cbb38aff-d0e4-425c-aa35-66313ff2002f/volumes"
Sep 16 19:06:57.270183 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:06:57.270148    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:07:32.512352 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:32.512318    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"]
Sep 16 19:07:43.288325 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:43.288288    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:07:47.531645 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.531596    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw" podUID="859538bc-95a2-41a6-9e35-bb4fc9664338" containerName="main" containerID="cri-o://04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a" gracePeriod=45
Sep 16 19:07:47.801843 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.801819    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:07:47.830977 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.830950    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-model-cache\") pod \"859538bc-95a2-41a6-9e35-bb4fc9664338\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") "
Sep 16 19:07:47.831200 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.830986    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859538bc-95a2-41a6-9e35-bb4fc9664338-tls-certs\") pod \"859538bc-95a2-41a6-9e35-bb4fc9664338\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") "
Sep 16 19:07:47.831200 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831016    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-dshm\") pod \"859538bc-95a2-41a6-9e35-bb4fc9664338\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") "
Sep 16 19:07:47.831200 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831049    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-tmp-dir\") pod \"859538bc-95a2-41a6-9e35-bb4fc9664338\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") "
Sep 16 19:07:47.831200 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831087    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-home\") pod \"859538bc-95a2-41a6-9e35-bb4fc9664338\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") "
Sep 16 19:07:47.831200 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831109    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltxg7\" (UniqueName: \"kubernetes.io/projected/859538bc-95a2-41a6-9e35-bb4fc9664338-kube-api-access-ltxg7\") pod \"859538bc-95a2-41a6-9e35-bb4fc9664338\" (UID: \"859538bc-95a2-41a6-9e35-bb4fc9664338\") "
Sep 16 19:07:47.831432 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831245    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-model-cache" (OuterVolumeSpecName: "model-cache") pod "859538bc-95a2-41a6-9e35-bb4fc9664338" (UID: "859538bc-95a2-41a6-9e35-bb4fc9664338"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:47.831432 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831342    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "859538bc-95a2-41a6-9e35-bb4fc9664338" (UID: "859538bc-95a2-41a6-9e35-bb4fc9664338"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:47.831548 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831534    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-home" (OuterVolumeSpecName: "home") pod "859538bc-95a2-41a6-9e35-bb4fc9664338" (UID: "859538bc-95a2-41a6-9e35-bb4fc9664338"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:47.831855 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831832    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-model-cache\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:07:47.831974 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831858    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-tmp-dir\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:07:47.831974 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.831872    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-home\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:07:47.833495 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.833420    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859538bc-95a2-41a6-9e35-bb4fc9664338-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "859538bc-95a2-41a6-9e35-bb4fc9664338" (UID: "859538bc-95a2-41a6-9e35-bb4fc9664338"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:47.833796 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.833503    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-dshm" (OuterVolumeSpecName: "dshm") pod "859538bc-95a2-41a6-9e35-bb4fc9664338" (UID: "859538bc-95a2-41a6-9e35-bb4fc9664338"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:47.833796 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.833564    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859538bc-95a2-41a6-9e35-bb4fc9664338-kube-api-access-ltxg7" (OuterVolumeSpecName: "kube-api-access-ltxg7") pod "859538bc-95a2-41a6-9e35-bb4fc9664338" (UID: "859538bc-95a2-41a6-9e35-bb4fc9664338"). InnerVolumeSpecName "kube-api-access-ltxg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:47.932543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.932515    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/859538bc-95a2-41a6-9e35-bb4fc9664338-tls-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:07:47.932543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.932539    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/859538bc-95a2-41a6-9e35-bb4fc9664338-dshm\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:07:47.932543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.932552    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltxg7\" (UniqueName: \"kubernetes.io/projected/859538bc-95a2-41a6-9e35-bb4fc9664338-kube-api-access-ltxg7\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:07:47.962455 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.962421    2575 generic.go:356] "Generic (PLEG): container finished" podID="859538bc-95a2-41a6-9e35-bb4fc9664338" containerID="04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a" exitCode=0
Sep 16 19:07:47.962613 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.962465    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"
Sep 16 19:07:47.962613 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.962512    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw" event={"ID":"859538bc-95a2-41a6-9e35-bb4fc9664338","Type":"ContainerDied","Data":"04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a"}
Sep 16 19:07:47.962613 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.962563    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw" event={"ID":"859538bc-95a2-41a6-9e35-bb4fc9664338","Type":"ContainerDied","Data":"610f9b9b7facf9bae67124a76e307499574e72c43d127efdb18b5c33c5e45085"}
Sep 16 19:07:47.962613 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.962581    2575 scope.go:117] "RemoveContainer" containerID="04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a"
Sep 16 19:07:47.972448 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.972430    2575 scope.go:117] "RemoveContainer" containerID="04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a"
Sep 16 19:07:47.972704 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:07:47.972686    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a\": container with ID starting with 04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a not found: ID does not exist" containerID="04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a"
Sep 16 19:07:47.972800 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.972729    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a"} err="failed to get container status \"04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a\": rpc error: code = NotFound desc = could not find container \"04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a\": container with ID starting with 04760e394b1dedaf19a414c7fdd6deaf547a568987422da419e99bc15beb426a not found: ID does not exist"
Sep 16 19:07:47.991298 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.991273    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"]
Sep 16 19:07:47.993520 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:47.993499    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/e2e-conflict-a-05590707-kserve-85d75df898-xkwpw"]
Sep 16 19:07:48.621360 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:48.621324    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="859538bc-95a2-41a6-9e35-bb4fc9664338" path="/var/lib/kubelet/pods/859538bc-95a2-41a6-9e35-bb4fc9664338/volumes"
Sep 16 19:07:52.898029 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.897993    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"]
Sep 16 19:07:52.898505 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.898492    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb38aff-d0e4-425c-aa35-66313ff2002f" containerName="main"
Sep 16 19:07:52.898505 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.898506    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb38aff-d0e4-425c-aa35-66313ff2002f" containerName="main"
Sep 16 19:07:52.898597 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.898517    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="859538bc-95a2-41a6-9e35-bb4fc9664338" containerName="main"
Sep 16 19:07:52.898597 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.898522    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="859538bc-95a2-41a6-9e35-bb4fc9664338" containerName="main"
Sep 16 19:07:52.898672 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.898604    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb38aff-d0e4-425c-aa35-66313ff2002f" containerName="main"
Sep 16 19:07:52.898672 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.898612    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="859538bc-95a2-41a6-9e35-bb4fc9664338" containerName="main"
Sep 16 19:07:52.903217 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.903193    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:52.907125 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.907099    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"distinct-w4-c090c8-kserve-self-signed-certs\""
Sep 16 19:07:52.927660 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.927635    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"]
Sep 16 19:07:52.985304 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.985265    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-model-cache\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:52.985479 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.985410    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-home\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:52.985479 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.985434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-dshm\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:52.985578 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.985484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-tmp-dir\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:52.985578 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.985507    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j6k5\" (UniqueName: \"kubernetes.io/projected/97659b56-0e7a-4321-82f1-9383bf2d46e8-kube-api-access-5j6k5\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:52.985578 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:52.985573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97659b56-0e7a-4321-82f1-9383bf2d46e8-tls-certs\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.086402 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.086367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-tmp-dir\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.086584 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.086408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j6k5\" (UniqueName: \"kubernetes.io/projected/97659b56-0e7a-4321-82f1-9383bf2d46e8-kube-api-access-5j6k5\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.086584 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.086444    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97659b56-0e7a-4321-82f1-9383bf2d46e8-tls-certs\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.086584 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.086531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-model-cache\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.086810 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.086648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-home\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.086810 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.086672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-dshm\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.086926 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.086880    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-tmp-dir\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.087029 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.087005    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-model-cache\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.087474 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.087040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-home\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.089147 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.089123    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-dshm\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.089261 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.089232    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97659b56-0e7a-4321-82f1-9383bf2d46e8-tls-certs\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.097317 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.097297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j6k5\" (UniqueName: \"kubernetes.io/projected/97659b56-0e7a-4321-82f1-9383bf2d46e8-kube-api-access-5j6k5\") pod \"distinct-w4-c090c8-kserve-5456598db6-7dktd\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.213772 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.213657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:07:53.495895 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:53.495861    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"]
Sep 16 19:07:53.497392 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:07:53.497367    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97659b56_0e7a_4321_82f1_9383bf2d46e8.slice/crio-85440a6376535440b6a37ae7eadddc8935bc91aab72e0148da5fe26f8f02145a WatchSource:0}: Error finding container 85440a6376535440b6a37ae7eadddc8935bc91aab72e0148da5fe26f8f02145a: Status 404 returned error can't find the container with id 85440a6376535440b6a37ae7eadddc8935bc91aab72e0148da5fe26f8f02145a
Sep 16 19:07:54.001439 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:54.001407    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd" event={"ID":"97659b56-0e7a-4321-82f1-9383bf2d46e8","Type":"ContainerStarted","Data":"731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db"}
Sep 16 19:07:54.001826 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:54.001448    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd" event={"ID":"97659b56-0e7a-4321-82f1-9383bf2d46e8","Type":"ContainerStarted","Data":"85440a6376535440b6a37ae7eadddc8935bc91aab72e0148da5fe26f8f02145a"}
Sep 16 19:07:54.023213 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:07:54.023166    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd" podStartSLOduration=2.023148625 podStartE2EDuration="2.023148625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:54.020561515 +0000 UTC m=+1701.837327101" watchObservedRunningTime="2026-09-16 19:07:54.023148625 +0000 UTC m=+1701.839914212"
Sep 16 19:08:03.214516 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:03.214423    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:08:03.214516 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:03.214478    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:08:03.227579 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:03.227543    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:08:04.071586 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:04.071553    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:08:07.709750 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:07.709688    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"]
Sep 16 19:08:07.710229 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:07.709916    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v" podUID="13ca0c09-5f94-4a43-8374-fff629695027" containerName="maas-api" containerID="cri-o://db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3" gracePeriod=30
Sep 16 19:08:07.979585 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:07.979523    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:08:08.046878 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.046845    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ca0c09-5f94-4a43-8374-fff629695027-openshift-service-ca\") pod \"13ca0c09-5f94-4a43-8374-fff629695027\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") "
Sep 16 19:08:08.047073 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.046905    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-metrics-certs\") pod \"13ca0c09-5f94-4a43-8374-fff629695027\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") "
Sep 16 19:08:08.047073 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.046947    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-maas-api-tls\") pod \"13ca0c09-5f94-4a43-8374-fff629695027\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") "
Sep 16 19:08:08.047312 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.047276    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ca0c09-5f94-4a43-8374-fff629695027-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13ca0c09-5f94-4a43-8374-fff629695027" (UID: "13ca0c09-5f94-4a43-8374-fff629695027"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:08.047449 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.047289    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g65hw\" (UniqueName: \"kubernetes.io/projected/13ca0c09-5f94-4a43-8374-fff629695027-kube-api-access-g65hw\") pod \"13ca0c09-5f94-4a43-8374-fff629695027\" (UID: \"13ca0c09-5f94-4a43-8374-fff629695027\") "
Sep 16 19:08:08.047845 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.047821    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ca0c09-5f94-4a43-8374-fff629695027-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:08:08.049536 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.049505    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ca0c09-5f94-4a43-8374-fff629695027-kube-api-access-g65hw" (OuterVolumeSpecName: "kube-api-access-g65hw") pod "13ca0c09-5f94-4a43-8374-fff629695027" (UID: "13ca0c09-5f94-4a43-8374-fff629695027"). InnerVolumeSpecName "kube-api-access-g65hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:08.049634 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.049612    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13ca0c09-5f94-4a43-8374-fff629695027" (UID: "13ca0c09-5f94-4a43-8374-fff629695027"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:08.050219 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.050201    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "13ca0c09-5f94-4a43-8374-fff629695027" (UID: "13ca0c09-5f94-4a43-8374-fff629695027"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:08.084913 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.084877    2575 generic.go:356] "Generic (PLEG): container finished" podID="13ca0c09-5f94-4a43-8374-fff629695027" containerID="db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3" exitCode=0
Sep 16 19:08:08.085049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.084923    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"
Sep 16 19:08:08.085049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.084926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v" event={"ID":"13ca0c09-5f94-4a43-8374-fff629695027","Type":"ContainerDied","Data":"db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3"}
Sep 16 19:08:08.085049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.084994    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v" event={"ID":"13ca0c09-5f94-4a43-8374-fff629695027","Type":"ContainerDied","Data":"4f56cde4cbf55776a8ca6654a665fc9b4ccb034c2cea70b094bc124f03a6f330"}
Sep 16 19:08:08.085049 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.085027    2575 scope.go:117] "RemoveContainer" containerID="db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3"
Sep 16 19:08:08.095041 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.095024    2575 scope.go:117] "RemoveContainer" containerID="db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3"
Sep 16 19:08:08.095298 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:08:08.095278    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3\": container with ID starting with db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3 not found: ID does not exist" containerID="db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3"
Sep 16 19:08:08.095369 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.095309    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3"} err="failed to get container status \"db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3\": rpc error: code = NotFound desc = could not find container \"db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3\": container with ID starting with db4e3e4b04b3d2102a64c0dbb610458fb73e4a718f71453f32af1ab46ad893c3 not found: ID does not exist"
Sep 16 19:08:08.114017 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.113989    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"]
Sep 16 19:08:08.117748 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.117724    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d929c607-9ccbf997c-v7c9v"]
Sep 16 19:08:08.149034 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.149004    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g65hw\" (UniqueName: \"kubernetes.io/projected/13ca0c09-5f94-4a43-8374-fff629695027-kube-api-access-g65hw\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:08:08.149034 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.149030    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:08:08.149034 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.149041    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13ca0c09-5f94-4a43-8374-fff629695027-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:08:08.622568 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:08.622536    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13ca0c09-5f94-4a43-8374-fff629695027" path="/var/lib/kubelet/pods/13ca0c09-5f94-4a43-8374-fff629695027/volumes"
Sep 16 19:08:19.696982 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:19.696947    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:08:54.015802 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:08:54.015765    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:09:20.931122 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:09:20.931087    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:09:54.722932 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:09:54.722894    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:10:13.886759 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.886724    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"]
Sep 16 19:10:13.887224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.887210    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ca0c09-5f94-4a43-8374-fff629695027" containerName="maas-api"
Sep 16 19:10:13.887270 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.887225    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ca0c09-5f94-4a43-8374-fff629695027" containerName="maas-api"
Sep 16 19:10:13.887311 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.887303    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ca0c09-5f94-4a43-8374-fff629695027" containerName="maas-api"
Sep 16 19:10:13.890467 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.890449    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:13.893471 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.893438    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a46230-serving-cert\""
Sep 16 19:10:13.938453 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.938429    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"]
Sep 16 19:10:13.964007 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.963977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-maas-api-tls\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:13.964187 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.964020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-metrics-certs\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:13.964187 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.964163    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-openshift-service-ca\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:13.964310 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:13.964229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtq5q\" (UniqueName: \"kubernetes.io/projected/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-kube-api-access-wtq5q\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.065617 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.065581    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-maas-api-tls\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.065617 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.065619    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-metrics-certs\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.065867 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.065680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-openshift-service-ca\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.065867 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:10:14.065762    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-serving-cert: secret "maas-api-e2e-api-a-a46230-serving-cert" not found
Sep 16 19:10:14.065867 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:10:14.065827    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-maas-api-tls podName:4cdff30c-5e21-44a5-9dd9-89db0a173a1b nodeName:}" failed. No retries permitted until 2026-09-16 19:10:14.565807882 +0000 UTC m=+1842.382573463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-maas-api-tls") pod "maas-api-e2e-api-a-a46230-c9b95c55b-6tm55" (UID: "4cdff30c-5e21-44a5-9dd9-89db0a173a1b") : secret "maas-api-e2e-api-a-a46230-serving-cert" not found
Sep 16 19:10:14.065867 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.065765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtq5q\" (UniqueName: \"kubernetes.io/projected/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-kube-api-access-wtq5q\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.066299 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.066277    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-openshift-service-ca\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.068238 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.068208    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-metrics-certs\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.087085 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.087056    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtq5q\" (UniqueName: \"kubernetes.io/projected/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-kube-api-access-wtq5q\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.570780 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.570741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-maas-api-tls\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.573221 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.573193    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-maas-api-tls\") pod \"maas-api-e2e-api-a-a46230-c9b95c55b-6tm55\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:14.800461 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:14.800434    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:15.090861 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:15.090833    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"]
Sep 16 19:10:15.093085 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:10:15.093046    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdff30c_5e21_44a5_9dd9_89db0a173a1b.slice/crio-22f7429af9564bb7e0bd1f694e359f291ec83d26c24cb6586b04273b6ca009d2 WatchSource:0}: Error finding container 22f7429af9564bb7e0bd1f694e359f291ec83d26c24cb6586b04273b6ca009d2: Status 404 returned error can't find the container with id 22f7429af9564bb7e0bd1f694e359f291ec83d26c24cb6586b04273b6ca009d2
Sep 16 19:10:15.094522 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:15.094499    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:10:15.815311 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:15.815273    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55" event={"ID":"4cdff30c-5e21-44a5-9dd9-89db0a173a1b","Type":"ContainerStarted","Data":"a216ab60e1e04eed08b63c2fdc5f68d17afd37f7cfc5bad78fc430e13996565d"}
Sep 16 19:10:15.815470 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:15.815327    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55" event={"ID":"4cdff30c-5e21-44a5-9dd9-89db0a173a1b","Type":"ContainerStarted","Data":"22f7429af9564bb7e0bd1f694e359f291ec83d26c24cb6586b04273b6ca009d2"}
Sep 16 19:10:15.815470 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:15.815410    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:15.836196 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:15.836148    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55" podStartSLOduration=2.244762743 podStartE2EDuration="2.836132791s" totalImagesPullingTime="591.370048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:15.833820945 +0000 UTC m=+1843.650586530" watchObservedRunningTime="2026-09-16 19:10:15.836132791 +0000 UTC m=+1843.652898376"
Sep 16 19:10:21.828528 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:21.828486    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:10:33.907055 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:33.907014    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"]
Sep 16 19:10:33.910869 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:33.910850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:33.913649 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:33.913627    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b04e50-serving-cert\""
Sep 16 19:10:33.923534 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:33.923512    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"]
Sep 16 19:10:33.975969 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:33.975939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqhm\" (UniqueName: \"kubernetes.io/projected/941f2848-b345-4f42-81fd-210ea44d0d37-kube-api-access-fpqhm\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:33.976117 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:33.976055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-metrics-certs\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:33.976161 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:33.976147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941f2848-b345-4f42-81fd-210ea44d0d37-openshift-service-ca\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:33.976197 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:33.976183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-maas-api-tls\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.076758 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.076697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941f2848-b345-4f42-81fd-210ea44d0d37-openshift-service-ca\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.076935 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.076787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-maas-api-tls\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.076935 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.076828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqhm\" (UniqueName: \"kubernetes.io/projected/941f2848-b345-4f42-81fd-210ea44d0d37-kube-api-access-fpqhm\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.076935 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.076917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-metrics-certs\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.077120 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:10:34.076925    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-serving-cert: secret "maas-api-e2e-api-b-b04e50-serving-cert" not found
Sep 16 19:10:34.077120 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:10:34.077004    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-maas-api-tls podName:941f2848-b345-4f42-81fd-210ea44d0d37 nodeName:}" failed. No retries permitted until 2026-09-16 19:10:34.57698379 +0000 UTC m=+1862.393749354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-maas-api-tls") pod "maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r" (UID: "941f2848-b345-4f42-81fd-210ea44d0d37") : secret "maas-api-e2e-api-b-b04e50-serving-cert" not found
Sep 16 19:10:34.077393 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.077373    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941f2848-b345-4f42-81fd-210ea44d0d37-openshift-service-ca\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.079392 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.079369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-metrics-certs\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.086819 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.086799    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqhm\" (UniqueName: \"kubernetes.io/projected/941f2848-b345-4f42-81fd-210ea44d0d37-kube-api-access-fpqhm\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.582464 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.582425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-maas-api-tls\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.584911 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.584887    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-maas-api-tls\") pod \"maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.726223 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.726189    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"]
Sep 16 19:10:34.731126 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.731100    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:34.734212 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.734164    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b04e50-dockercfg-kv7xw\""
Sep 16 19:10:34.734212 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.734202    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b04e50\""
Sep 16 19:10:34.741971 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.741894    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"]
Sep 16 19:10:34.784966 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.784937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeccf0f-2658-48b4-948c-e539b01d52cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw\" (UID: \"fbeccf0f-2658-48b4-948c-e539b01d52cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:34.785129 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.784982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twf4\" (UniqueName: \"kubernetes.io/projected/fbeccf0f-2658-48b4-948c-e539b01d52cd-kube-api-access-8twf4\") pod \"payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw\" (UID: \"fbeccf0f-2658-48b4-948c-e539b01d52cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:34.824823 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.824792    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:34.886874 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.886838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeccf0f-2658-48b4-948c-e539b01d52cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw\" (UID: \"fbeccf0f-2658-48b4-948c-e539b01d52cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:34.886996 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.886907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twf4\" (UniqueName: \"kubernetes.io/projected/fbeccf0f-2658-48b4-948c-e539b01d52cd-kube-api-access-8twf4\") pod \"payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw\" (UID: \"fbeccf0f-2658-48b4-948c-e539b01d52cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:34.888198 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.888167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeccf0f-2658-48b4-948c-e539b01d52cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw\" (UID: \"fbeccf0f-2658-48b4-948c-e539b01d52cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:34.897157 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:34.897135    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twf4\" (UniqueName: \"kubernetes.io/projected/fbeccf0f-2658-48b4-948c-e539b01d52cd-kube-api-access-8twf4\") pod \"payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw\" (UID: \"fbeccf0f-2658-48b4-948c-e539b01d52cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:35.045379 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.045337    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:35.109875 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.109681    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"]
Sep 16 19:10:35.328223 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.328197    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"]
Sep 16 19:10:35.330636 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:10:35.330605    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbeccf0f_2658_48b4_948c_e539b01d52cd.slice/crio-13d9d655f4242d5f3b7e2e77ce35efb51ec1cc063ebbb686e372deb01ce08023 WatchSource:0}: Error finding container 13d9d655f4242d5f3b7e2e77ce35efb51ec1cc063ebbb686e372deb01ce08023: Status 404 returned error can't find the container with id 13d9d655f4242d5f3b7e2e77ce35efb51ec1cc063ebbb686e372deb01ce08023
Sep 16 19:10:35.945813 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.945773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r" event={"ID":"941f2848-b345-4f42-81fd-210ea44d0d37","Type":"ContainerStarted","Data":"a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6"}
Sep 16 19:10:35.946024 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.945839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r" event={"ID":"941f2848-b345-4f42-81fd-210ea44d0d37","Type":"ContainerStarted","Data":"880591ede3fc77b3dc498107afad5f77531892115918e4b73dab3281e74c203e"}
Sep 16 19:10:35.946024 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.945931    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:35.947529 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.947499    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw" event={"ID":"fbeccf0f-2658-48b4-948c-e539b01d52cd","Type":"ContainerStarted","Data":"3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212"}
Sep 16 19:10:35.947746 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.947549    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw" event={"ID":"fbeccf0f-2658-48b4-948c-e539b01d52cd","Type":"ContainerStarted","Data":"13d9d655f4242d5f3b7e2e77ce35efb51ec1cc063ebbb686e372deb01ce08023"}
Sep 16 19:10:35.947746 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.947581    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:35.970665 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.970609    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r" podStartSLOduration=2.634800194 podStartE2EDuration="2.970592191s" totalImagesPullingTime="335.791997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:35.9672415 +0000 UTC m=+1863.784007088" watchObservedRunningTime="2026-09-16 19:10:35.970592191 +0000 UTC m=+1863.787357777"
Sep 16 19:10:35.989189 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:35.989128    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw" podStartSLOduration=1.989111296 podStartE2EDuration="1.989111296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:35.987056837 +0000 UTC m=+1863.803822449" watchObservedRunningTime="2026-09-16 19:10:35.989111296 +0000 UTC m=+1863.805876882"
Sep 16 19:10:40.911288 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:40.911254    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"]
Sep 16 19:10:40.911796 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:40.911448    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7" podUID="f34a3c69-14d6-4d29-bc3a-711647a01b5a" containerName="maas-api" containerID="cri-o://6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde" gracePeriod=30
Sep 16 19:10:40.965135 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:10:40.965097    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34a3c69_14d6_4d29_bc3a_711647a01b5a.slice/crio-6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:10:41.180769 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.180699    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 19:10:41.255044 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.255010    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-maas-api-tls\") pod \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") "
Sep 16 19:10:41.255231 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.255065    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-metrics-certs\") pod \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") "
Sep 16 19:10:41.255231 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.255171    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9sv6\" (UniqueName: \"kubernetes.io/projected/f34a3c69-14d6-4d29-bc3a-711647a01b5a-kube-api-access-h9sv6\") pod \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") "
Sep 16 19:10:41.255231 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.255196    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34a3c69-14d6-4d29-bc3a-711647a01b5a-openshift-service-ca\") pod \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\" (UID: \"f34a3c69-14d6-4d29-bc3a-711647a01b5a\") "
Sep 16 19:10:41.255602 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.255573    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34a3c69-14d6-4d29-bc3a-711647a01b5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f34a3c69-14d6-4d29-bc3a-711647a01b5a" (UID: "f34a3c69-14d6-4d29-bc3a-711647a01b5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:41.257324 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.257297    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f34a3c69-14d6-4d29-bc3a-711647a01b5a" (UID: "f34a3c69-14d6-4d29-bc3a-711647a01b5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:41.257763 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.257738    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f34a3c69-14d6-4d29-bc3a-711647a01b5a" (UID: "f34a3c69-14d6-4d29-bc3a-711647a01b5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:41.257834 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.257765    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34a3c69-14d6-4d29-bc3a-711647a01b5a-kube-api-access-h9sv6" (OuterVolumeSpecName: "kube-api-access-h9sv6") pod "f34a3c69-14d6-4d29-bc3a-711647a01b5a" (UID: "f34a3c69-14d6-4d29-bc3a-711647a01b5a"). InnerVolumeSpecName "kube-api-access-h9sv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:41.356787 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.356743    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.356787 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.356786    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34a3c69-14d6-4d29-bc3a-711647a01b5a-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.356999 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.356799    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9sv6\" (UniqueName: \"kubernetes.io/projected/f34a3c69-14d6-4d29-bc3a-711647a01b5a-kube-api-access-h9sv6\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.356999 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.356812    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34a3c69-14d6-4d29-bc3a-711647a01b5a-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.959198 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.959169    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:10:41.984466 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.984435    2575 generic.go:356] "Generic (PLEG): container finished" podID="f34a3c69-14d6-4d29-bc3a-711647a01b5a" containerID="6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde" exitCode=0
Sep 16 19:10:41.984666 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.984482    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"
Sep 16 19:10:41.984666 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.984492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7" event={"ID":"f34a3c69-14d6-4d29-bc3a-711647a01b5a","Type":"ContainerDied","Data":"6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde"}
Sep 16 19:10:41.984666 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.984545    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7" event={"ID":"f34a3c69-14d6-4d29-bc3a-711647a01b5a","Type":"ContainerDied","Data":"ec8c1c2592c4ddad229947ace6d6db62b726ae7db60835ac42453cc39de81885"}
Sep 16 19:10:41.984666 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.984573    2575 scope.go:117] "RemoveContainer" containerID="6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde"
Sep 16 19:10:41.995120 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.995099    2575 scope.go:117] "RemoveContainer" containerID="6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde"
Sep 16 19:10:41.995437 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:10:41.995412    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde\": container with ID starting with 6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde not found: ID does not exist" containerID="6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde"
Sep 16 19:10:41.995508 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:41.995445    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde"} err="failed to get container status \"6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde\": rpc error: code = NotFound desc = could not find container \"6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde\": container with ID starting with 6316086ea4e8fef1b1de2a43619e48942233c58584c640b5e0f838f13eaa3bde not found: ID does not exist"
Sep 16 19:10:42.056365 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:42.056322    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"]
Sep 16 19:10:42.069399 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:42.069369    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1857c-867b496f8-jqvr7"]
Sep 16 19:10:42.621885 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:42.621846    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f34a3c69-14d6-4d29-bc3a-711647a01b5a" path="/var/lib/kubelet/pods/f34a3c69-14d6-4d29-bc3a-711647a01b5a/volumes"
Sep 16 19:10:46.955499 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:46.955467    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:10:54.585110 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:10:54.585075    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:11:09.941746 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:09.941698    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"]
Sep 16 19:11:09.942155 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:09.941919    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55" podUID="4cdff30c-5e21-44a5-9dd9-89db0a173a1b" containerName="maas-api" containerID="cri-o://a216ab60e1e04eed08b63c2fdc5f68d17afd37f7cfc5bad78fc430e13996565d" gracePeriod=30
Sep 16 19:11:10.165754 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.165693    2575 generic.go:356] "Generic (PLEG): container finished" podID="4cdff30c-5e21-44a5-9dd9-89db0a173a1b" containerID="a216ab60e1e04eed08b63c2fdc5f68d17afd37f7cfc5bad78fc430e13996565d" exitCode=0
Sep 16 19:11:10.165754 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.165741    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55" event={"ID":"4cdff30c-5e21-44a5-9dd9-89db0a173a1b","Type":"ContainerDied","Data":"a216ab60e1e04eed08b63c2fdc5f68d17afd37f7cfc5bad78fc430e13996565d"}
Sep 16 19:11:10.205256 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.205194    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:11:10.267491 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.267448    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtq5q\" (UniqueName: \"kubernetes.io/projected/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-kube-api-access-wtq5q\") pod \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") "
Sep 16 19:11:10.267491 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.267497    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-metrics-certs\") pod \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") "
Sep 16 19:11:10.267834 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.267645    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-openshift-service-ca\") pod \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") "
Sep 16 19:11:10.267834 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.267764    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-maas-api-tls\") pod \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\" (UID: \"4cdff30c-5e21-44a5-9dd9-89db0a173a1b\") "
Sep 16 19:11:10.268051 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.268023    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4cdff30c-5e21-44a5-9dd9-89db0a173a1b" (UID: "4cdff30c-5e21-44a5-9dd9-89db0a173a1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:10.268217 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.268201    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.269941 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.269914    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-kube-api-access-wtq5q" (OuterVolumeSpecName: "kube-api-access-wtq5q") pod "4cdff30c-5e21-44a5-9dd9-89db0a173a1b" (UID: "4cdff30c-5e21-44a5-9dd9-89db0a173a1b"). InnerVolumeSpecName "kube-api-access-wtq5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:10.270023 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.269916    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4cdff30c-5e21-44a5-9dd9-89db0a173a1b" (UID: "4cdff30c-5e21-44a5-9dd9-89db0a173a1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:10.270023 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.269941    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4cdff30c-5e21-44a5-9dd9-89db0a173a1b" (UID: "4cdff30c-5e21-44a5-9dd9-89db0a173a1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:10.369361 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.369324    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.369361 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.369358    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtq5q\" (UniqueName: \"kubernetes.io/projected/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-kube-api-access-wtq5q\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.369361 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:10.369368    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdff30c-5e21-44a5-9dd9-89db0a173a1b-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.175096 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:11.175068    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"
Sep 16 19:11:11.175642 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:11.175065    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55" event={"ID":"4cdff30c-5e21-44a5-9dd9-89db0a173a1b","Type":"ContainerDied","Data":"22f7429af9564bb7e0bd1f694e359f291ec83d26c24cb6586b04273b6ca009d2"}
Sep 16 19:11:11.175642 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:11.175226    2575 scope.go:117] "RemoveContainer" containerID="a216ab60e1e04eed08b63c2fdc5f68d17afd37f7cfc5bad78fc430e13996565d"
Sep 16 19:11:11.200084 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:11.200043    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"]
Sep 16 19:11:11.202397 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:11.202368    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a46230-c9b95c55b-6tm55"]
Sep 16 19:11:12.622245 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:12.622209    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cdff30c-5e21-44a5-9dd9-89db0a173a1b" path="/var/lib/kubelet/pods/4cdff30c-5e21-44a5-9dd9-89db0a173a1b/volumes"
Sep 16 19:11:30.708332 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:30.708295    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:11:34.872949 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:34.872915    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"]
Sep 16 19:11:34.873378 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:34.873169    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r" podUID="941f2848-b345-4f42-81fd-210ea44d0d37" containerName="maas-api" containerID="cri-o://a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6" gracePeriod=30
Sep 16 19:11:34.948065 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:34.948027    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"]
Sep 16 19:11:34.948301 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:34.948274    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw" podUID="fbeccf0f-2658-48b4-948c-e539b01d52cd" containerName="payload-pre-processing" containerID="cri-o://3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212" gracePeriod=30
Sep 16 19:11:35.203125 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.203100    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:11:35.236520 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.236493    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:11:35.236910 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.236889    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpqhm\" (UniqueName: \"kubernetes.io/projected/941f2848-b345-4f42-81fd-210ea44d0d37-kube-api-access-fpqhm\") pod \"941f2848-b345-4f42-81fd-210ea44d0d37\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") "
Sep 16 19:11:35.237071 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.236963    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941f2848-b345-4f42-81fd-210ea44d0d37-openshift-service-ca\") pod \"941f2848-b345-4f42-81fd-210ea44d0d37\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") "
Sep 16 19:11:35.237164 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.237111    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-maas-api-tls\") pod \"941f2848-b345-4f42-81fd-210ea44d0d37\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") "
Sep 16 19:11:35.237164 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.237157    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-metrics-certs\") pod \"941f2848-b345-4f42-81fd-210ea44d0d37\" (UID: \"941f2848-b345-4f42-81fd-210ea44d0d37\") "
Sep 16 19:11:35.237360 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.237328    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941f2848-b345-4f42-81fd-210ea44d0d37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "941f2848-b345-4f42-81fd-210ea44d0d37" (UID: "941f2848-b345-4f42-81fd-210ea44d0d37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:35.237570 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.237554    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941f2848-b345-4f42-81fd-210ea44d0d37-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:35.239239 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.239215    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941f2848-b345-4f42-81fd-210ea44d0d37-kube-api-access-fpqhm" (OuterVolumeSpecName: "kube-api-access-fpqhm") pod "941f2848-b345-4f42-81fd-210ea44d0d37" (UID: "941f2848-b345-4f42-81fd-210ea44d0d37"). InnerVolumeSpecName "kube-api-access-fpqhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:35.239239 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.239225    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "941f2848-b345-4f42-81fd-210ea44d0d37" (UID: "941f2848-b345-4f42-81fd-210ea44d0d37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:35.239363 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.239313    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "941f2848-b345-4f42-81fd-210ea44d0d37" (UID: "941f2848-b345-4f42-81fd-210ea44d0d37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:35.320028 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.319991    2575 generic.go:356] "Generic (PLEG): container finished" podID="941f2848-b345-4f42-81fd-210ea44d0d37" containerID="a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6" exitCode=0
Sep 16 19:11:35.320224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.320042    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"
Sep 16 19:11:35.320224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.320078    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r" event={"ID":"941f2848-b345-4f42-81fd-210ea44d0d37","Type":"ContainerDied","Data":"a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6"}
Sep 16 19:11:35.320224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.320148    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r" event={"ID":"941f2848-b345-4f42-81fd-210ea44d0d37","Type":"ContainerDied","Data":"880591ede3fc77b3dc498107afad5f77531892115918e4b73dab3281e74c203e"}
Sep 16 19:11:35.320224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.320184    2575 scope.go:117] "RemoveContainer" containerID="a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6"
Sep 16 19:11:35.321691 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.321667    2575 generic.go:356] "Generic (PLEG): container finished" podID="fbeccf0f-2658-48b4-948c-e539b01d52cd" containerID="3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212" exitCode=0
Sep 16 19:11:35.321828 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.321702    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw" event={"ID":"fbeccf0f-2658-48b4-948c-e539b01d52cd","Type":"ContainerDied","Data":"3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212"}
Sep 16 19:11:35.321828 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.321769    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw" event={"ID":"fbeccf0f-2658-48b4-948c-e539b01d52cd","Type":"ContainerDied","Data":"13d9d655f4242d5f3b7e2e77ce35efb51ec1cc063ebbb686e372deb01ce08023"}
Sep 16 19:11:35.321828 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.321729    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"
Sep 16 19:11:35.332012 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.331984    2575 scope.go:117] "RemoveContainer" containerID="a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6"
Sep 16 19:11:35.332314 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:11:35.332294    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6\": container with ID starting with a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6 not found: ID does not exist" containerID="a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6"
Sep 16 19:11:35.332362 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.332323    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6"} err="failed to get container status \"a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6\": rpc error: code = NotFound desc = could not find container \"a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6\": container with ID starting with a10d68e599ad5c68a1e133c91d90629564638261e77940c14346da26c5d873a6 not found: ID does not exist"
Sep 16 19:11:35.332362 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.332341    2575 scope.go:117] "RemoveContainer" containerID="3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212"
Sep 16 19:11:35.338659 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.338632    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeccf0f-2658-48b4-948c-e539b01d52cd-plugins-config-volume\") pod \"fbeccf0f-2658-48b4-948c-e539b01d52cd\" (UID: \"fbeccf0f-2658-48b4-948c-e539b01d52cd\") "
Sep 16 19:11:35.338805 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.338736    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8twf4\" (UniqueName: \"kubernetes.io/projected/fbeccf0f-2658-48b4-948c-e539b01d52cd-kube-api-access-8twf4\") pod \"fbeccf0f-2658-48b4-948c-e539b01d52cd\" (UID: \"fbeccf0f-2658-48b4-948c-e539b01d52cd\") "
Sep 16 19:11:35.339202 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.339172    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbeccf0f-2658-48b4-948c-e539b01d52cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fbeccf0f-2658-48b4-948c-e539b01d52cd" (UID: "fbeccf0f-2658-48b4-948c-e539b01d52cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:35.339317 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.339184    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqhm\" (UniqueName: \"kubernetes.io/projected/941f2848-b345-4f42-81fd-210ea44d0d37-kube-api-access-fpqhm\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:35.339317 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.339238    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:35.339317 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.339257    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941f2848-b345-4f42-81fd-210ea44d0d37-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:35.340966 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.340942    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbeccf0f-2658-48b4-948c-e539b01d52cd-kube-api-access-8twf4" (OuterVolumeSpecName: "kube-api-access-8twf4") pod "fbeccf0f-2658-48b4-948c-e539b01d52cd" (UID: "fbeccf0f-2658-48b4-948c-e539b01d52cd"). InnerVolumeSpecName "kube-api-access-8twf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:35.341278 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.341258    2575 scope.go:117] "RemoveContainer" containerID="3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212"
Sep 16 19:11:35.341593 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:11:35.341576    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212\": container with ID starting with 3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212 not found: ID does not exist" containerID="3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212"
Sep 16 19:11:35.341657 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.341601    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212"} err="failed to get container status \"3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212\": rpc error: code = NotFound desc = could not find container \"3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212\": container with ID starting with 3515fda2082a1d751fad607699b358178306f29015c9c5218bec1bf00390c212 not found: ID does not exist"
Sep 16 19:11:35.374232 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.374194    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"]
Sep 16 19:11:35.388867 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.388403    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b04e50-6c599df6c9-tzk7r"]
Sep 16 19:11:35.440775 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.440641    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeccf0f-2658-48b4-948c-e539b01d52cd-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:35.440775 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.440676    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twf4\" (UniqueName: \"kubernetes.io/projected/fbeccf0f-2658-48b4-948c-e539b01d52cd-kube-api-access-8twf4\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:35.667017 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.666977    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"]
Sep 16 19:11:35.673558 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:35.673530    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b04e50-57dd588758-2cxvw"]
Sep 16 19:11:36.622069 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:36.622025    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="941f2848-b345-4f42-81fd-210ea44d0d37" path="/var/lib/kubelet/pods/941f2848-b345-4f42-81fd-210ea44d0d37/volumes"
Sep 16 19:11:36.622655 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:36.622611    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbeccf0f-2658-48b4-948c-e539b01d52cd" path="/var/lib/kubelet/pods/fbeccf0f-2658-48b4-948c-e539b01d52cd/volumes"
Sep 16 19:11:54.706796 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:54.706763    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"]
Sep 16 19:11:54.710738 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:54.707498    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4" podUID="3c44466b-d290-4ae9-88c4-38797cf872da" containerName="maas-api" containerID="cri-o://113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9" gracePeriod=30
Sep 16 19:11:54.812194 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:54.812149    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"]
Sep 16 19:11:54.812504 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:54.812450    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv" podUID="1b98ec98-39eb-4ed5-8ae1-641cf6093c9e" containerName="payload-pre-processing" containerID="cri-o://0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8" gracePeriod=30
Sep 16 19:11:54.974000 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:54.973940    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:11:55.058487 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.058457    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:11:55.145428 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.145395    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-metrics-certs\") pod \"3c44466b-d290-4ae9-88c4-38797cf872da\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") "
Sep 16 19:11:55.145595 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.145444    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-maas-api-tls\") pod \"3c44466b-d290-4ae9-88c4-38797cf872da\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") "
Sep 16 19:11:55.145595 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.145493    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6tcw\" (UniqueName: \"kubernetes.io/projected/3c44466b-d290-4ae9-88c4-38797cf872da-kube-api-access-w6tcw\") pod \"3c44466b-d290-4ae9-88c4-38797cf872da\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") "
Sep 16 19:11:55.145763 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.145740    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c44466b-d290-4ae9-88c4-38797cf872da-openshift-service-ca\") pod \"3c44466b-d290-4ae9-88c4-38797cf872da\" (UID: \"3c44466b-d290-4ae9-88c4-38797cf872da\") "
Sep 16 19:11:55.146134 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.146097    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c44466b-d290-4ae9-88c4-38797cf872da-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c44466b-d290-4ae9-88c4-38797cf872da" (UID: "3c44466b-d290-4ae9-88c4-38797cf872da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:55.147580 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.147552    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3c44466b-d290-4ae9-88c4-38797cf872da" (UID: "3c44466b-d290-4ae9-88c4-38797cf872da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:55.147580 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.147572    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c44466b-d290-4ae9-88c4-38797cf872da" (UID: "3c44466b-d290-4ae9-88c4-38797cf872da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:55.147703 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.147601    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c44466b-d290-4ae9-88c4-38797cf872da-kube-api-access-w6tcw" (OuterVolumeSpecName: "kube-api-access-w6tcw") pod "3c44466b-d290-4ae9-88c4-38797cf872da" (UID: "3c44466b-d290-4ae9-88c4-38797cf872da"). InnerVolumeSpecName "kube-api-access-w6tcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:55.247134 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.247058    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-plugins-config-volume\") pod \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\" (UID: \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\") "
Sep 16 19:11:55.247266 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.247160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgfsf\" (UniqueName: \"kubernetes.io/projected/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-kube-api-access-xgfsf\") pod \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\" (UID: \"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e\") "
Sep 16 19:11:55.247428 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.247405    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b98ec98-39eb-4ed5-8ae1-641cf6093c9e" (UID: "1b98ec98-39eb-4ed5-8ae1-641cf6093c9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:55.247476 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.247420    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:55.247476 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.247453    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c44466b-d290-4ae9-88c4-38797cf872da-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:55.247476 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.247464    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tcw\" (UniqueName: \"kubernetes.io/projected/3c44466b-d290-4ae9-88c4-38797cf872da-kube-api-access-w6tcw\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:55.247476 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.247473    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c44466b-d290-4ae9-88c4-38797cf872da-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:55.249281 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.249259    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-kube-api-access-xgfsf" (OuterVolumeSpecName: "kube-api-access-xgfsf") pod "1b98ec98-39eb-4ed5-8ae1-641cf6093c9e" (UID: "1b98ec98-39eb-4ed5-8ae1-641cf6093c9e"). InnerVolumeSpecName "kube-api-access-xgfsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:55.349252 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.349210    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:55.349252 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.349248    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgfsf\" (UniqueName: \"kubernetes.io/projected/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e-kube-api-access-xgfsf\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:11:55.433844 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.433812    2575 generic.go:356] "Generic (PLEG): container finished" podID="3c44466b-d290-4ae9-88c4-38797cf872da" containerID="113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9" exitCode=0
Sep 16 19:11:55.434034 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.433863    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"
Sep 16 19:11:55.434034 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.433912    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4" event={"ID":"3c44466b-d290-4ae9-88c4-38797cf872da","Type":"ContainerDied","Data":"113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9"}
Sep 16 19:11:55.434034 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.433965    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4" event={"ID":"3c44466b-d290-4ae9-88c4-38797cf872da","Type":"ContainerDied","Data":"40a18150ec8bbf8a570bccc58c791e776bee66784da4b1da0091e6b80c0d761d"}
Sep 16 19:11:55.434034 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.433993    2575 scope.go:117] "RemoveContainer" containerID="113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9"
Sep 16 19:11:55.435491 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.435466    2575 generic.go:356] "Generic (PLEG): container finished" podID="1b98ec98-39eb-4ed5-8ae1-641cf6093c9e" containerID="0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8" exitCode=0
Sep 16 19:11:55.435593 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.435504    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"
Sep 16 19:11:55.435593 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.435549    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv" event={"ID":"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e","Type":"ContainerDied","Data":"0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8"}
Sep 16 19:11:55.435688 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.435596    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv" event={"ID":"1b98ec98-39eb-4ed5-8ae1-641cf6093c9e","Type":"ContainerDied","Data":"1be982908b5b5143807938e8234c1aa816c9fe25409e10f4facfe22e4662e909"}
Sep 16 19:11:55.444540 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.444520    2575 scope.go:117] "RemoveContainer" containerID="113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9"
Sep 16 19:11:55.445008 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:11:55.444978    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9\": container with ID starting with 113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9 not found: ID does not exist" containerID="113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9"
Sep 16 19:11:55.445117 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.445012    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9"} err="failed to get container status \"113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9\": rpc error: code = NotFound desc = could not find container \"113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9\": container with ID starting with 113908799ee5a3013cbc4bcc4dcdfb676e1f192020c5129280b41c98bdd62fe9 not found: ID does not exist"
Sep 16 19:11:55.445117 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.445036    2575 scope.go:117] "RemoveContainer" containerID="0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8"
Sep 16 19:11:55.454399 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.454380    2575 scope.go:117] "RemoveContainer" containerID="0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8"
Sep 16 19:11:55.454696 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:11:55.454677    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8\": container with ID starting with 0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8 not found: ID does not exist" containerID="0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8"
Sep 16 19:11:55.454855 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.454704    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8"} err="failed to get container status \"0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8\": rpc error: code = NotFound desc = could not find container \"0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8\": container with ID starting with 0f3094b63e36daa30fffef3fa37fb15ff2a2b1566fe7f2976cb19e3e362b0fb8 not found: ID does not exist"
Sep 16 19:11:55.464829 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.464793    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"]
Sep 16 19:11:55.468107 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.468083    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c20219-6d4f5f99c8-bqnbv"]
Sep 16 19:11:55.479068 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.479044    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"]
Sep 16 19:11:55.483362 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:55.483337    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c20219-779c4dd464-fcfb4"]
Sep 16 19:11:56.527894 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:56.527854    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:11:56.621027 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:56.620990    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b98ec98-39eb-4ed5-8ae1-641cf6093c9e" path="/var/lib/kubelet/pods/1b98ec98-39eb-4ed5-8ae1-641cf6093c9e/volumes"
Sep 16 19:11:56.621422 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:11:56.621409    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c44466b-d290-4ae9-88c4-38797cf872da" path="/var/lib/kubelet/pods/3c44466b-d290-4ae9-88c4-38797cf872da/volumes"
Sep 16 19:12:14.100193 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.100150    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"]
Sep 16 19:12:14.100831 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.100376    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk" podUID="e9598949-c34c-4b41-9ff8-0bec5d90df35" containerName="maas-api" containerID="cri-o://e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862" gracePeriod=30
Sep 16 19:12:14.178316 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.178279    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"]
Sep 16 19:12:14.178608 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.178553    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf" podUID="72425a16-256c-4a76-bf68-33032cbc96a8" containerName="payload-pre-processing" containerID="cri-o://7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2" gracePeriod=30
Sep 16 19:12:14.435371 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.435344    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:12:14.469060 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.469033    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:12:14.543045 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.543017    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9598949-c34c-4b41-9ff8-0bec5d90df35-openshift-service-ca\") pod \"e9598949-c34c-4b41-9ff8-0bec5d90df35\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") "
Sep 16 19:12:14.543222 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.543107    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-metrics-certs\") pod \"e9598949-c34c-4b41-9ff8-0bec5d90df35\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") "
Sep 16 19:12:14.543222 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.543130    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-maas-api-tls\") pod \"e9598949-c34c-4b41-9ff8-0bec5d90df35\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") "
Sep 16 19:12:14.543222 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.543184    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52wtm\" (UniqueName: \"kubernetes.io/projected/e9598949-c34c-4b41-9ff8-0bec5d90df35-kube-api-access-52wtm\") pod \"e9598949-c34c-4b41-9ff8-0bec5d90df35\" (UID: \"e9598949-c34c-4b41-9ff8-0bec5d90df35\") "
Sep 16 19:12:14.543222 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.543215    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkv6v\" (UniqueName: \"kubernetes.io/projected/72425a16-256c-4a76-bf68-33032cbc96a8-kube-api-access-mkv6v\") pod \"72425a16-256c-4a76-bf68-33032cbc96a8\" (UID: \"72425a16-256c-4a76-bf68-33032cbc96a8\") "
Sep 16 19:12:14.543402 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.543274    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72425a16-256c-4a76-bf68-33032cbc96a8-plugins-config-volume\") pod \"72425a16-256c-4a76-bf68-33032cbc96a8\" (UID: \"72425a16-256c-4a76-bf68-33032cbc96a8\") "
Sep 16 19:12:14.543693 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.543661    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72425a16-256c-4a76-bf68-33032cbc96a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72425a16-256c-4a76-bf68-33032cbc96a8" (UID: "72425a16-256c-4a76-bf68-33032cbc96a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:14.543933 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.543888    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9598949-c34c-4b41-9ff8-0bec5d90df35-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9598949-c34c-4b41-9ff8-0bec5d90df35" (UID: "e9598949-c34c-4b41-9ff8-0bec5d90df35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:14.545786 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.545727    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72425a16-256c-4a76-bf68-33032cbc96a8-kube-api-access-mkv6v" (OuterVolumeSpecName: "kube-api-access-mkv6v") pod "72425a16-256c-4a76-bf68-33032cbc96a8" (UID: "72425a16-256c-4a76-bf68-33032cbc96a8"). InnerVolumeSpecName "kube-api-access-mkv6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:14.545786 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.545737    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9598949-c34c-4b41-9ff8-0bec5d90df35" (UID: "e9598949-c34c-4b41-9ff8-0bec5d90df35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:14.546086 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.546069    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9598949-c34c-4b41-9ff8-0bec5d90df35" (UID: "e9598949-c34c-4b41-9ff8-0bec5d90df35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:14.546251 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.546230    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9598949-c34c-4b41-9ff8-0bec5d90df35-kube-api-access-52wtm" (OuterVolumeSpecName: "kube-api-access-52wtm") pod "e9598949-c34c-4b41-9ff8-0bec5d90df35" (UID: "e9598949-c34c-4b41-9ff8-0bec5d90df35"). InnerVolumeSpecName "kube-api-access-52wtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:14.549492 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.549470    2575 generic.go:356] "Generic (PLEG): container finished" podID="72425a16-256c-4a76-bf68-33032cbc96a8" containerID="7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2" exitCode=0
Sep 16 19:12:14.549594 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.549519    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"
Sep 16 19:12:14.549657 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.549515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf" event={"ID":"72425a16-256c-4a76-bf68-33032cbc96a8","Type":"ContainerDied","Data":"7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2"}
Sep 16 19:12:14.549755 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.549652    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf" event={"ID":"72425a16-256c-4a76-bf68-33032cbc96a8","Type":"ContainerDied","Data":"122db46634159b02bc135e541da74047aeb8db960ba9de6dc06a356a447cca87"}
Sep 16 19:12:14.549755 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.549691    2575 scope.go:117] "RemoveContainer" containerID="7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2"
Sep 16 19:12:14.551045 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.551025    2575 generic.go:356] "Generic (PLEG): container finished" podID="e9598949-c34c-4b41-9ff8-0bec5d90df35" containerID="e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862" exitCode=0
Sep 16 19:12:14.551139 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.551052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk" event={"ID":"e9598949-c34c-4b41-9ff8-0bec5d90df35","Type":"ContainerDied","Data":"e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862"}
Sep 16 19:12:14.551139 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.551067    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"
Sep 16 19:12:14.551139 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.551085    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk" event={"ID":"e9598949-c34c-4b41-9ff8-0bec5d90df35","Type":"ContainerDied","Data":"00b04fb57da427d5c1f3e84f31d5423e735effd1df15981cda34d92eb16802e3"}
Sep 16 19:12:14.558826 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.558806    2575 scope.go:117] "RemoveContainer" containerID="7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2"
Sep 16 19:12:14.559129 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:12:14.559109    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2\": container with ID starting with 7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2 not found: ID does not exist" containerID="7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2"
Sep 16 19:12:14.559216 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.559141    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2"} err="failed to get container status \"7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2\": rpc error: code = NotFound desc = could not find container \"7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2\": container with ID starting with 7fece695d68473d9ad09df09e3388933ba315014f2456efcffc604ea14f49bf2 not found: ID does not exist"
Sep 16 19:12:14.559216 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.559175    2575 scope.go:117] "RemoveContainer" containerID="e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862"
Sep 16 19:12:14.567539 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.567516    2575 scope.go:117] "RemoveContainer" containerID="e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862"
Sep 16 19:12:14.567838 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:12:14.567819    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862\": container with ID starting with e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862 not found: ID does not exist" containerID="e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862"
Sep 16 19:12:14.567905 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.567844    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862"} err="failed to get container status \"e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862\": rpc error: code = NotFound desc = could not find container \"e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862\": container with ID starting with e3c112de2fcc876274932137e19d0978c20667a98674bad5275974c570e13862 not found: ID does not exist"
Sep 16 19:12:14.580853 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.580832    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"]
Sep 16 19:12:14.584532 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.584510    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58bf51-75854c4dd9-sj7pk"]
Sep 16 19:12:14.621171 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.621150    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9598949-c34c-4b41-9ff8-0bec5d90df35" path="/var/lib/kubelet/pods/e9598949-c34c-4b41-9ff8-0bec5d90df35/volumes"
Sep 16 19:12:14.621436 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.621417    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"]
Sep 16 19:12:14.625050 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.625028    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58bf51-694866656b-lqmgf"]
Sep 16 19:12:14.645038 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.644992    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:14.645038 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.645011    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9598949-c34c-4b41-9ff8-0bec5d90df35-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:14.645038 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.645021    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52wtm\" (UniqueName: \"kubernetes.io/projected/e9598949-c34c-4b41-9ff8-0bec5d90df35-kube-api-access-52wtm\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:14.645038 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.645030    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkv6v\" (UniqueName: \"kubernetes.io/projected/72425a16-256c-4a76-bf68-33032cbc96a8-kube-api-access-mkv6v\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:14.645038 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.645040    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72425a16-256c-4a76-bf68-33032cbc96a8-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:14.645240 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:14.645050    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9598949-c34c-4b41-9ff8-0bec5d90df35-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:16.621338 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:16.621301    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72425a16-256c-4a76-bf68-33032cbc96a8" path="/var/lib/kubelet/pods/72425a16-256c-4a76-bf68-33032cbc96a8/volumes"
Sep 16 19:12:22.311539 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:22.311501    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:12:33.770794 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:33.770760    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"]
Sep 16 19:12:33.788688 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:33.788658    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"]
Sep 16 19:12:45.082499 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.082463    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"]
Sep 16 19:12:45.082965 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.082686    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb" podUID="18b8a36e-c095-4321-9878-04b47e7b7072" containerName="maas-api" containerID="cri-o://a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d" gracePeriod=30
Sep 16 19:12:45.159609 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.159571    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"]
Sep 16 19:12:45.159897 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.159799    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw" podUID="24c67554-1901-4306-b10c-99602e43c9bc" containerName="payload-pre-processing" containerID="cri-o://63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200" gracePeriod=30
Sep 16 19:12:45.307088 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.307050    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:12:45.431216 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.431184    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 19:12:45.456074 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.456046    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 19:12:45.547823 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.547792    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c67554-1901-4306-b10c-99602e43c9bc-plugins-config-volume\") pod \"24c67554-1901-4306-b10c-99602e43c9bc\" (UID: \"24c67554-1901-4306-b10c-99602e43c9bc\") "
Sep 16 19:12:45.548010 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.547841    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-metrics-certs\") pod \"18b8a36e-c095-4321-9878-04b47e7b7072\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") "
Sep 16 19:12:45.548010 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.547878    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrk5z\" (UniqueName: \"kubernetes.io/projected/18b8a36e-c095-4321-9878-04b47e7b7072-kube-api-access-hrk5z\") pod \"18b8a36e-c095-4321-9878-04b47e7b7072\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") "
Sep 16 19:12:45.548010 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.547909    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnpcp\" (UniqueName: \"kubernetes.io/projected/24c67554-1901-4306-b10c-99602e43c9bc-kube-api-access-xnpcp\") pod \"24c67554-1901-4306-b10c-99602e43c9bc\" (UID: \"24c67554-1901-4306-b10c-99602e43c9bc\") "
Sep 16 19:12:45.548010 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.547927    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8a36e-c095-4321-9878-04b47e7b7072-openshift-service-ca\") pod \"18b8a36e-c095-4321-9878-04b47e7b7072\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") "
Sep 16 19:12:45.548224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.548034    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-maas-api-tls\") pod \"18b8a36e-c095-4321-9878-04b47e7b7072\" (UID: \"18b8a36e-c095-4321-9878-04b47e7b7072\") "
Sep 16 19:12:45.548291 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.548228    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c67554-1901-4306-b10c-99602e43c9bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24c67554-1901-4306-b10c-99602e43c9bc" (UID: "24c67554-1901-4306-b10c-99602e43c9bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:45.548376 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.548355    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c67554-1901-4306-b10c-99602e43c9bc-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.548437 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.548412    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b8a36e-c095-4321-9878-04b47e7b7072-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18b8a36e-c095-4321-9878-04b47e7b7072" (UID: "18b8a36e-c095-4321-9878-04b47e7b7072"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:45.550221 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.550190    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b8a36e-c095-4321-9878-04b47e7b7072-kube-api-access-hrk5z" (OuterVolumeSpecName: "kube-api-access-hrk5z") pod "18b8a36e-c095-4321-9878-04b47e7b7072" (UID: "18b8a36e-c095-4321-9878-04b47e7b7072"). InnerVolumeSpecName "kube-api-access-hrk5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:45.550343 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.550226    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "18b8a36e-c095-4321-9878-04b47e7b7072" (UID: "18b8a36e-c095-4321-9878-04b47e7b7072"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:45.550343 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.550231    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c67554-1901-4306-b10c-99602e43c9bc-kube-api-access-xnpcp" (OuterVolumeSpecName: "kube-api-access-xnpcp") pod "24c67554-1901-4306-b10c-99602e43c9bc" (UID: "24c67554-1901-4306-b10c-99602e43c9bc"). InnerVolumeSpecName "kube-api-access-xnpcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:45.550343 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.550244    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18b8a36e-c095-4321-9878-04b47e7b7072" (UID: "18b8a36e-c095-4321-9878-04b47e7b7072"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:45.649283 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.649199    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.649283 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.649229    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18b8a36e-c095-4321-9878-04b47e7b7072-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.649283 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.649239    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrk5z\" (UniqueName: \"kubernetes.io/projected/18b8a36e-c095-4321-9878-04b47e7b7072-kube-api-access-hrk5z\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.649283 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.649255    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnpcp\" (UniqueName: \"kubernetes.io/projected/24c67554-1901-4306-b10c-99602e43c9bc-kube-api-access-xnpcp\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.649283 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.649268    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b8a36e-c095-4321-9878-04b47e7b7072-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.719356 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.719320    2575 generic.go:356] "Generic (PLEG): container finished" podID="18b8a36e-c095-4321-9878-04b47e7b7072" containerID="a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d" exitCode=0
Sep 16 19:12:45.719538 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.719367    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"
Sep 16 19:12:45.719538 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.719399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb" event={"ID":"18b8a36e-c095-4321-9878-04b47e7b7072","Type":"ContainerDied","Data":"a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d"}
Sep 16 19:12:45.719538 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.719449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb" event={"ID":"18b8a36e-c095-4321-9878-04b47e7b7072","Type":"ContainerDied","Data":"45542a474bdca088ab08729c74107110f2d731b506ad2eceb647245973f59767"}
Sep 16 19:12:45.719538 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.719475    2575 scope.go:117] "RemoveContainer" containerID="a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d"
Sep 16 19:12:45.720944 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.720922    2575 generic.go:356] "Generic (PLEG): container finished" podID="24c67554-1901-4306-b10c-99602e43c9bc" containerID="63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200" exitCode=0
Sep 16 19:12:45.721062 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.720989    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"
Sep 16 19:12:45.721062 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.721001    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw" event={"ID":"24c67554-1901-4306-b10c-99602e43c9bc","Type":"ContainerDied","Data":"63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200"}
Sep 16 19:12:45.721062 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.721043    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw" event={"ID":"24c67554-1901-4306-b10c-99602e43c9bc","Type":"ContainerDied","Data":"7b9614cd6003abf03a68b410cb0ac8df30ba77a48c494412976317d17c424e32"}
Sep 16 19:12:45.731505 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.731485    2575 scope.go:117] "RemoveContainer" containerID="a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d"
Sep 16 19:12:45.731949 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:12:45.731824    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d\": container with ID starting with a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d not found: ID does not exist" containerID="a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d"
Sep 16 19:12:45.731949 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.731851    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d"} err="failed to get container status \"a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d\": rpc error: code = NotFound desc = could not find container \"a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d\": container with ID starting with a0380964718dd084d94fd8a8f14071773c3809fa4e0f7151e103f99bdd97d30d not found: ID does not exist"
Sep 16 19:12:45.731949 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.731873    2575 scope.go:117] "RemoveContainer" containerID="63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200"
Sep 16 19:12:45.740453 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.740432    2575 scope.go:117] "RemoveContainer" containerID="63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200"
Sep 16 19:12:45.740751 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:12:45.740721    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200\": container with ID starting with 63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200 not found: ID does not exist" containerID="63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200"
Sep 16 19:12:45.740813 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.740757    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200"} err="failed to get container status \"63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200\": rpc error: code = NotFound desc = could not find container \"63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200\": container with ID starting with 63af249bbdc4b14492e511c29770f2a060d21d7e349e0a7d7ecf4a1f6374b200 not found: ID does not exist"
Sep 16 19:12:45.748854 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.748815    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"]
Sep 16 19:12:45.751017 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.750997    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c090c8-5cc5fd99f7-vp9fb"]
Sep 16 19:12:45.763506 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.763478    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"]
Sep 16 19:12:45.768148 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:45.768125    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c090c8-664f795fbb-652vw"]
Sep 16 19:12:46.621817 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:46.621780    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18b8a36e-c095-4321-9878-04b47e7b7072" path="/var/lib/kubelet/pods/18b8a36e-c095-4321-9878-04b47e7b7072/volumes"
Sep 16 19:12:46.622210 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:46.622171    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c67554-1901-4306-b10c-99602e43c9bc" path="/var/lib/kubelet/pods/24c67554-1901-4306-b10c-99602e43c9bc/volumes"
Sep 16 19:12:48.790659 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:48.790618    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd" podUID="97659b56-0e7a-4321-82f1-9383bf2d46e8" containerName="main" containerID="cri-o://731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db" gracePeriod=45
Sep 16 19:12:48.807925 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:48.807888    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt" podUID="8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" containerName="main" containerID="cri-o://62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d" gracePeriod=45
Sep 16 19:12:49.082149 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.082122    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:12:49.085044 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.085024    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:12:49.182946 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.182905    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-dshm\") pod \"97659b56-0e7a-4321-82f1-9383bf2d46e8\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") "
Sep 16 19:12:49.183110 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.182978    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-model-cache\") pod \"97659b56-0e7a-4321-82f1-9383bf2d46e8\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") "
Sep 16 19:12:49.183110 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183011    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-model-cache\") pod \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") "
Sep 16 19:12:49.183110 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183045    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-home\") pod \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") "
Sep 16 19:12:49.183110 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183097    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-tmp-dir\") pod \"97659b56-0e7a-4321-82f1-9383bf2d46e8\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") "
Sep 16 19:12:49.183328 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183123    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tls-certs\") pod \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") "
Sep 16 19:12:49.183328 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183157    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97659b56-0e7a-4321-82f1-9383bf2d46e8-tls-certs\") pod \"97659b56-0e7a-4321-82f1-9383bf2d46e8\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") "
Sep 16 19:12:49.183328 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183193    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-dshm\") pod \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") "
Sep 16 19:12:49.183328 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183223    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j6k5\" (UniqueName: \"kubernetes.io/projected/97659b56-0e7a-4321-82f1-9383bf2d46e8-kube-api-access-5j6k5\") pod \"97659b56-0e7a-4321-82f1-9383bf2d46e8\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") "
Sep 16 19:12:49.183328 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183267    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzhzq\" (UniqueName: \"kubernetes.io/projected/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-kube-api-access-tzhzq\") pod \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") "
Sep 16 19:12:49.183328 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183290    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-home\") pod \"97659b56-0e7a-4321-82f1-9383bf2d46e8\" (UID: \"97659b56-0e7a-4321-82f1-9383bf2d46e8\") "
Sep 16 19:12:49.183328 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183307    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "97659b56-0e7a-4321-82f1-9383bf2d46e8" (UID: "97659b56-0e7a-4321-82f1-9383bf2d46e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.183687 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183305    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" (UID: "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.183687 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183344    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tmp-dir\") pod \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\" (UID: \"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce\") "
Sep 16 19:12:49.183687 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183552    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97659b56-0e7a-4321-82f1-9383bf2d46e8" (UID: "97659b56-0e7a-4321-82f1-9383bf2d46e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.183687 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183672    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-home" (OuterVolumeSpecName: "home") pod "97659b56-0e7a-4321-82f1-9383bf2d46e8" (UID: "97659b56-0e7a-4321-82f1-9383bf2d46e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.183919 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183786    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" (UID: "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.183978 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183946    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-home\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.183978 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183968    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tmp-dir\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.184080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183983    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-model-cache\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.184080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.183997    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-model-cache\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.184080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.184009    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-tmp-dir\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.184238 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.184089    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-home" (OuterVolumeSpecName: "home") pod "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" (UID: "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.185746 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.185682    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97659b56-0e7a-4321-82f1-9383bf2d46e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97659b56-0e7a-4321-82f1-9383bf2d46e8" (UID: "97659b56-0e7a-4321-82f1-9383bf2d46e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:49.186642 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.186618    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97659b56-0e7a-4321-82f1-9383bf2d46e8-kube-api-access-5j6k5" (OuterVolumeSpecName: "kube-api-access-5j6k5") pod "97659b56-0e7a-4321-82f1-9383bf2d46e8" (UID: "97659b56-0e7a-4321-82f1-9383bf2d46e8"). InnerVolumeSpecName "kube-api-access-5j6k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:49.186942 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.186922    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-dshm" (OuterVolumeSpecName: "dshm") pod "97659b56-0e7a-4321-82f1-9383bf2d46e8" (UID: "97659b56-0e7a-4321-82f1-9383bf2d46e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.187007 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.186938    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-kube-api-access-tzhzq" (OuterVolumeSpecName: "kube-api-access-tzhzq") pod "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" (UID: "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce"). InnerVolumeSpecName "kube-api-access-tzhzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:49.187132 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.187116    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-dshm" (OuterVolumeSpecName: "dshm") pod "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" (UID: "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.187340 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.187326    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" (UID: "8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:49.285389 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.285359    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-tls-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.285389 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.285383    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97659b56-0e7a-4321-82f1-9383bf2d46e8-tls-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.285535 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.285396    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-dshm\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.285535 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.285407    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j6k5\" (UniqueName: \"kubernetes.io/projected/97659b56-0e7a-4321-82f1-9383bf2d46e8-kube-api-access-5j6k5\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.285535 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.285417    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhzq\" (UniqueName: \"kubernetes.io/projected/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-kube-api-access-tzhzq\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.285535 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.285425    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97659b56-0e7a-4321-82f1-9383bf2d46e8-dshm\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.285535 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.285434    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce-home\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.746275 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.746239    2575 generic.go:356] "Generic (PLEG): container finished" podID="97659b56-0e7a-4321-82f1-9383bf2d46e8" containerID="731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db" exitCode=0
Sep 16 19:12:49.746448 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.746284    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"
Sep 16 19:12:49.746448 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.746318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd" event={"ID":"97659b56-0e7a-4321-82f1-9383bf2d46e8","Type":"ContainerDied","Data":"731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db"}
Sep 16 19:12:49.746448 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.746361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd" event={"ID":"97659b56-0e7a-4321-82f1-9383bf2d46e8","Type":"ContainerDied","Data":"85440a6376535440b6a37ae7eadddc8935bc91aab72e0148da5fe26f8f02145a"}
Sep 16 19:12:49.746448 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.746385    2575 scope.go:117] "RemoveContainer" containerID="731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db"
Sep 16 19:12:49.747665 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.747647    2575 generic.go:356] "Generic (PLEG): container finished" podID="8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" containerID="62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d" exitCode=0
Sep 16 19:12:49.747777 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.747743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt" event={"ID":"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce","Type":"ContainerDied","Data":"62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d"}
Sep 16 19:12:49.747777 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.747757    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"
Sep 16 19:12:49.747777 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.747768    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt" event={"ID":"8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce","Type":"ContainerDied","Data":"7b77b9f44a51705f67d990d9114ca282927d54765b289191a19940ebb6d77200"}
Sep 16 19:12:49.757195 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.757171    2575 scope.go:117] "RemoveContainer" containerID="731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db"
Sep 16 19:12:49.757483 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:12:49.757464    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db\": container with ID starting with 731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db not found: ID does not exist" containerID="731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db"
Sep 16 19:12:49.757550 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.757490    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db"} err="failed to get container status \"731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db\": rpc error: code = NotFound desc = could not find container \"731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db\": container with ID starting with 731f4db6a71d40f439d8ffa52a8e7c7a971e216bb96d4b1be627072587c402db not found: ID does not exist"
Sep 16 19:12:49.757550 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.757508    2575 scope.go:117] "RemoveContainer" containerID="62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d"
Sep 16 19:12:49.765595 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.765576    2575 scope.go:117] "RemoveContainer" containerID="62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d"
Sep 16 19:12:49.765891 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:12:49.765873    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d\": container with ID starting with 62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d not found: ID does not exist" containerID="62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d"
Sep 16 19:12:49.765954 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.765898    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d"} err="failed to get container status \"62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d\": rpc error: code = NotFound desc = could not find container \"62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d\": container with ID starting with 62ee182d68eb2236344b95b2f5ad0fc6921e150b1684e75c9a9cb05be3c27f9d not found: ID does not exist"
Sep 16 19:12:49.776157 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.776134    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"]
Sep 16 19:12:49.780488 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.780463    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-w4-c090c8-kserve-5456598db6-7dktd"]
Sep 16 19:12:49.798173 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.798153    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"]
Sep 16 19:12:49.804202 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:49.804184    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/premium-w4-c090c8-kserve-6487f58547-2rbdt"]
Sep 16 19:12:50.621891 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:50.621858    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" path="/var/lib/kubelet/pods/8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce/volumes"
Sep 16 19:12:50.622288 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:12:50.622274    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97659b56-0e7a-4321-82f1-9383bf2d46e8" path="/var/lib/kubelet/pods/97659b56-0e7a-4321-82f1-9383bf2d46e8/volumes"
Sep 16 19:13:05.218651 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:13:05.218614    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:13:14.613070 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:13:14.613037    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:13:40.512341 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:13:40.512303    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:13:55.122139 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:13:55.122046    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:14:08.507193 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:14:08.507158    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:14:20.200666 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:14:20.200627    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:14:29.607407 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:14:29.607371    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:14:41.078865 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:14:41.078820    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:15:04.711147 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:04.711110    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:15:21.319778 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:21.319742    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:15:30.709285 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:30.709250    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:15:41.011626 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:41.011585    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:15:45.073595 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.073558    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"]
Sep 16 19:15:45.074080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074021    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9598949-c34c-4b41-9ff8-0bec5d90df35" containerName="maas-api"
Sep 16 19:15:45.074080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074035    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9598949-c34c-4b41-9ff8-0bec5d90df35" containerName="maas-api"
Sep 16 19:15:45.074080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074047    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97659b56-0e7a-4321-82f1-9383bf2d46e8" containerName="main"
Sep 16 19:15:45.074080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074053    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="97659b56-0e7a-4321-82f1-9383bf2d46e8" containerName="main"
Sep 16 19:15:45.074080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074062    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b8a36e-c095-4321-9878-04b47e7b7072" containerName="maas-api"
Sep 16 19:15:45.074080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074068    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b8a36e-c095-4321-9878-04b47e7b7072" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074086    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" containerName="main"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074092    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" containerName="main"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074097    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b98ec98-39eb-4ed5-8ae1-641cf6093c9e" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074103    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b98ec98-39eb-4ed5-8ae1-641cf6093c9e" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074127    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbeccf0f-2658-48b4-948c-e539b01d52cd" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074133    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbeccf0f-2658-48b4-948c-e539b01d52cd" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074141    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c44466b-d290-4ae9-88c4-38797cf872da" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074147    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c44466b-d290-4ae9-88c4-38797cf872da" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074155    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72425a16-256c-4a76-bf68-33032cbc96a8" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074160    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="72425a16-256c-4a76-bf68-33032cbc96a8" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074170    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34a3c69-14d6-4d29-bc3a-711647a01b5a" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074175    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34a3c69-14d6-4d29-bc3a-711647a01b5a" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074182    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdff30c-5e21-44a5-9dd9-89db0a173a1b" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074187    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdff30c-5e21-44a5-9dd9-89db0a173a1b" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074192    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c67554-1901-4306-b10c-99602e43c9bc" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074197    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c67554-1901-4306-b10c-99602e43c9bc" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074203    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941f2848-b345-4f42-81fd-210ea44d0d37" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074209    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="941f2848-b345-4f42-81fd-210ea44d0d37" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074278    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbeccf0f-2658-48b4-948c-e539b01d52cd" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074285    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdff30c-5e21-44a5-9dd9-89db0a173a1b" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074291    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="941f2848-b345-4f42-81fd-210ea44d0d37" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074297    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c67554-1901-4306-b10c-99602e43c9bc" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074303    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="72425a16-256c-4a76-bf68-33032cbc96a8" containerName="payload-pre-processing"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074308    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="97659b56-0e7a-4321-82f1-9383bf2d46e8" containerName="main"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074313    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9598949-c34c-4b41-9ff8-0bec5d90df35" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074319    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34a3c69-14d6-4d29-bc3a-711647a01b5a" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074324    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b8a36e-c095-4321-9878-04b47e7b7072" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074329    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed67a6f-8fbd-48c2-b8b4-70ae56ae86ce" containerName="main"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074334    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c44466b-d290-4ae9-88c4-38797cf872da" containerName="maas-api"
Sep 16 19:15:45.074406 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.074340    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b98ec98-39eb-4ed5-8ae1-641cf6093c9e" containerName="payload-pre-processing"
Sep 16 19:15:45.077497 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.077471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.079837 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.079813    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4992ef-serving-cert\""
Sep 16 19:15:45.088797 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.088770    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"]
Sep 16 19:15:45.174609 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.174571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4k4w\" (UniqueName: \"kubernetes.io/projected/bcca0525-447f-4048-a4ca-c5bab292d55f-kube-api-access-l4k4w\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.174803 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.174623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-metrics-certs\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.174803 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.174749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.174803 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.174779    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcca0525-447f-4048-a4ca-c5bab292d55f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.275892 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.275852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4k4w\" (UniqueName: \"kubernetes.io/projected/bcca0525-447f-4048-a4ca-c5bab292d55f-kube-api-access-l4k4w\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.276066 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.275903    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-metrics-certs\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.276066 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.275959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.276066 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.275983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcca0525-447f-4048-a4ca-c5bab292d55f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.276203 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:15:45.276094    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-serving-cert: secret "maas-api-e2e-ipp-a-4992ef-serving-cert" not found
Sep 16 19:15:45.276203 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:15:45.276170    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-maas-api-tls podName:bcca0525-447f-4048-a4ca-c5bab292d55f nodeName:}" failed. No retries permitted until 2026-09-16 19:15:45.776152 +0000 UTC m=+2173.592917578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-maas-api-tls") pod "maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk" (UID: "bcca0525-447f-4048-a4ca-c5bab292d55f") : secret "maas-api-e2e-ipp-a-4992ef-serving-cert" not found
Sep 16 19:15:45.276641 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.276621    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcca0525-447f-4048-a4ca-c5bab292d55f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.278395 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.278369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-metrics-certs\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.284831 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.284808    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4k4w\" (UniqueName: \"kubernetes.io/projected/bcca0525-447f-4048-a4ca-c5bab292d55f-kube-api-access-l4k4w\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.782140 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.782089    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.784954 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.784924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:45.990020 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:45.989982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:46.265919 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:46.265886    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"]
Sep 16 19:15:46.267575 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:15:46.267545    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcca0525_447f_4048_a4ca_c5bab292d55f.slice/crio-2d265a39bd81be5a78b3fe55224bfd2f034120dee0e9f5430830d0bca7dae709 WatchSource:0}: Error finding container 2d265a39bd81be5a78b3fe55224bfd2f034120dee0e9f5430830d0bca7dae709: Status 404 returned error can't find the container with id 2d265a39bd81be5a78b3fe55224bfd2f034120dee0e9f5430830d0bca7dae709
Sep 16 19:15:46.268983 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:46.268965    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:15:46.673899 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:46.673860    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk" event={"ID":"bcca0525-447f-4048-a4ca-c5bab292d55f","Type":"ContainerStarted","Data":"2d265a39bd81be5a78b3fe55224bfd2f034120dee0e9f5430830d0bca7dae709"}
Sep 16 19:15:47.681261 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:47.681228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk" event={"ID":"bcca0525-447f-4048-a4ca-c5bab292d55f","Type":"ContainerStarted","Data":"982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d"}
Sep 16 19:15:47.681730 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:47.681287    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:47.701275 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:47.701219    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk" podStartSLOduration=1.640493982 podStartE2EDuration="2.701200851s" totalImagesPullingTime="1.060706869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:47.699389334 +0000 UTC m=+2175.516154921" watchObservedRunningTime="2026-09-16 19:15:47.701200851 +0000 UTC m=+2175.517966438"
Sep 16 19:15:53.692472 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:53.692440    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:15:56.100959 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:15:56.100918    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:16:20.083652 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.083619    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"]
Sep 16 19:16:20.086416 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.086395    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.089627 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.089569    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-766919-serving-cert\""
Sep 16 19:16:20.104974 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.104863    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"]
Sep 16 19:16:20.112504 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.112478    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzdx\" (UniqueName: \"kubernetes.io/projected/77775d98-da83-4226-81ff-ea323737a5d1-kube-api-access-2fzdx\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.112610 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.112516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77775d98-da83-4226-81ff-ea323737a5d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.112610 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.112575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.112610 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.112596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-metrics-certs\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.214109 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.214078    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzdx\" (UniqueName: \"kubernetes.io/projected/77775d98-da83-4226-81ff-ea323737a5d1-kube-api-access-2fzdx\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.214261 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.214115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77775d98-da83-4226-81ff-ea323737a5d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.214261 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.214144    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.214261 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.214160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-metrics-certs\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.214261 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:16:20.214243    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-serving-cert: secret "maas-api-e2e-ipp-b-766919-serving-cert" not found
Sep 16 19:16:20.214424 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:16:20.214321    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-maas-api-tls podName:77775d98-da83-4226-81ff-ea323737a5d1 nodeName:}" failed. No retries permitted until 2026-09-16 19:16:20.714300546 +0000 UTC m=+2208.531066110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-maas-api-tls") pod "maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb" (UID: "77775d98-da83-4226-81ff-ea323737a5d1") : secret "maas-api-e2e-ipp-b-766919-serving-cert" not found
Sep 16 19:16:20.214857 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.214839    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77775d98-da83-4226-81ff-ea323737a5d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.216562 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.216545    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-metrics-certs\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.225407 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.225379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzdx\" (UniqueName: \"kubernetes.io/projected/77775d98-da83-4226-81ff-ea323737a5d1-kube-api-access-2fzdx\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.719774 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.719700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.722380 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.722355    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:20.998007 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:20.997912    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:21.279572 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:21.279543    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"]
Sep 16 19:16:21.282033 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:16:21.282010    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77775d98_da83_4226_81ff_ea323737a5d1.slice/crio-bd21ddf6589d124668a0ff92b439fa6fb5309ac12f34f184d0204ffcfcc588d7 WatchSource:0}: Error finding container bd21ddf6589d124668a0ff92b439fa6fb5309ac12f34f184d0204ffcfcc588d7: Status 404 returned error can't find the container with id bd21ddf6589d124668a0ff92b439fa6fb5309ac12f34f184d0204ffcfcc588d7
Sep 16 19:16:21.868163 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:21.868123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb" event={"ID":"77775d98-da83-4226-81ff-ea323737a5d1","Type":"ContainerStarted","Data":"3978a87ff838c1ac13bf05605fc1b7f10269f72386c442de5805618a1038dc35"}
Sep 16 19:16:21.868334 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:21.868193    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:16:21.868334 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:21.868214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb" event={"ID":"77775d98-da83-4226-81ff-ea323737a5d1","Type":"ContainerStarted","Data":"bd21ddf6589d124668a0ff92b439fa6fb5309ac12f34f184d0204ffcfcc588d7"}
Sep 16 19:16:21.889890 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:21.889839    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb" podStartSLOduration=1.5125563149999999 podStartE2EDuration="1.889821991s" totalImagesPullingTime="377.265676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:21.885782686 +0000 UTC m=+2209.702548272" watchObservedRunningTime="2026-09-16 19:16:21.889821991 +0000 UTC m=+2209.706587568"
Sep 16 19:16:23.313844 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:23.313809    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:16:27.879592 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:16:27.879561    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:17:04.637520 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:04.637444    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:17:17.816417 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:17.816380    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:17:30.601825 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.601781    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"]
Sep 16 19:17:30.606259 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.606228    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.609314 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.609288    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-312c48-serving-cert\""
Sep 16 19:17:30.622992 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.622959    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"]
Sep 16 19:17:30.710285 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.710244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zp7k\" (UniqueName: \"kubernetes.io/projected/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-kube-api-access-2zp7k\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.710465 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.710309    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.710465 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.710366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.710465 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.710420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.811457 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.811421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zp7k\" (UniqueName: \"kubernetes.io/projected/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-kube-api-access-2zp7k\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.811621 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.811487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.811621 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.811521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.811621 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.811548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.812124 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.812094    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.814086 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.814062    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.814229 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.814208    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.821468 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.821442    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zp7k\" (UniqueName: \"kubernetes.io/projected/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-kube-api-access-2zp7k\") pod \"maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:30.916585 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:30.916497    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:31.199523 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:31.199444    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"]
Sep 16 19:17:31.202759 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:17:31.202669    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d84f0a_c5d9_4abe_bbc2_85d7fd5e8983.slice/crio-59df3783be88ceaff15c6c0bf0734530f77e683a26f62d591d3eabf738ff0303 WatchSource:0}: Error finding container 59df3783be88ceaff15c6c0bf0734530f77e683a26f62d591d3eabf738ff0303: Status 404 returned error can't find the container with id 59df3783be88ceaff15c6c0bf0734530f77e683a26f62d591d3eabf738ff0303
Sep 16 19:17:31.245482 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:31.245446    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc" event={"ID":"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983","Type":"ContainerStarted","Data":"59df3783be88ceaff15c6c0bf0734530f77e683a26f62d591d3eabf738ff0303"}
Sep 16 19:17:32.253167 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:32.253129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc" event={"ID":"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983","Type":"ContainerStarted","Data":"6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3"}
Sep 16 19:17:32.253548 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:32.253198    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:17:32.275271 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:32.275218    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc" podStartSLOduration=1.846352333 podStartE2EDuration="2.27520259s" totalImagesPullingTime="428.850257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:32.273309199 +0000 UTC m=+2280.090074785" watchObservedRunningTime="2026-09-16 19:17:32.27520259 +0000 UTC m=+2280.091968175"
Sep 16 19:17:35.934738 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:35.934692    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:17:38.263551 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:17:38.263519    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:18:08.496411 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.496377    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"]
Sep 16 19:18:08.496903 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.496591    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc" podUID="50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" containerName="maas-api" containerID="cri-o://6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3" gracePeriod=30
Sep 16 19:18:08.755727 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.755648    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:18:08.885352 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.885321    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-metrics-certs\") pod \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") "
Sep 16 19:18:08.885352 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.885363    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-openshift-service-ca\") pod \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") "
Sep 16 19:18:08.885545 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.885480    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zp7k\" (UniqueName: \"kubernetes.io/projected/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-kube-api-access-2zp7k\") pod \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") "
Sep 16 19:18:08.885545 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.885500    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-maas-api-tls\") pod \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\" (UID: \"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983\") "
Sep 16 19:18:08.885840 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.885811    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" (UID: "50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:08.887452 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.887423    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" (UID: "50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:08.887575 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.887548    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-kube-api-access-2zp7k" (OuterVolumeSpecName: "kube-api-access-2zp7k") pod "50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" (UID: "50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983"). InnerVolumeSpecName "kube-api-access-2zp7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:08.887647 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.887607    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" (UID: "50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:08.986830 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.986795    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zp7k\" (UniqueName: \"kubernetes.io/projected/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-kube-api-access-2zp7k\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:18:08.986830 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.986824    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:18:08.986830 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.986837    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:18:08.987047 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:08.986849    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:18:09.457014 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:09.456982    2575 generic.go:356] "Generic (PLEG): container finished" podID="50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" containerID="6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3" exitCode=0
Sep 16 19:18:09.457210 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:09.457025    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"
Sep 16 19:18:09.457210 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:09.457067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc" event={"ID":"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983","Type":"ContainerDied","Data":"6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3"}
Sep 16 19:18:09.457210 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:09.457121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc" event={"ID":"50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983","Type":"ContainerDied","Data":"59df3783be88ceaff15c6c0bf0734530f77e683a26f62d591d3eabf738ff0303"}
Sep 16 19:18:09.457210 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:09.457146    2575 scope.go:117] "RemoveContainer" containerID="6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3"
Sep 16 19:18:09.467175 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:09.467155    2575 scope.go:117] "RemoveContainer" containerID="6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3"
Sep 16 19:18:09.467444 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:18:09.467426    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3\": container with ID starting with 6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3 not found: ID does not exist" containerID="6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3"
Sep 16 19:18:09.467488 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:09.467453    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3"} err="failed to get container status \"6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3\": rpc error: code = NotFound desc = could not find container \"6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3\": container with ID starting with 6268a3af64d3048240510d161879f565903cfc2b6265ac207ae85944eade55d3 not found: ID does not exist"
Sep 16 19:18:09.480350 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:09.480329    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"]
Sep 16 19:18:09.489328 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:09.487253    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-312c48-7cd5dc9fc5-g7dxc"]
Sep 16 19:18:10.621591 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:10.621548    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" path="/var/lib/kubelet/pods/50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983/volumes"
Sep 16 19:18:13.021202 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:13.021156    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:18:32.911569 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:32.911486    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:18:35.246399 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.246361    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"]
Sep 16 19:18:35.246873 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.246644    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk" podUID="bcca0525-447f-4048-a4ca-c5bab292d55f" containerName="maas-api" containerID="cri-o://982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d" gracePeriod=30
Sep 16 19:18:35.505648 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.505576    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:18:35.596192 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.596160    2575 generic.go:356] "Generic (PLEG): container finished" podID="bcca0525-447f-4048-a4ca-c5bab292d55f" containerID="982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d" exitCode=0
Sep 16 19:18:35.596369 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.596219    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"
Sep 16 19:18:35.596369 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.596239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk" event={"ID":"bcca0525-447f-4048-a4ca-c5bab292d55f","Type":"ContainerDied","Data":"982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d"}
Sep 16 19:18:35.596369 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.596297    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk" event={"ID":"bcca0525-447f-4048-a4ca-c5bab292d55f","Type":"ContainerDied","Data":"2d265a39bd81be5a78b3fe55224bfd2f034120dee0e9f5430830d0bca7dae709"}
Sep 16 19:18:35.596369 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.596327    2575 scope.go:117] "RemoveContainer" containerID="982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d"
Sep 16 19:18:35.607298 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.607270    2575 scope.go:117] "RemoveContainer" containerID="982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d"
Sep 16 19:18:35.607678 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:18:35.607653    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d\": container with ID starting with 982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d not found: ID does not exist" containerID="982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d"
Sep 16 19:18:35.607785 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.607692    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d"} err="failed to get container status \"982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d\": rpc error: code = NotFound desc = could not find container \"982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d\": container with ID starting with 982ea0555b0cc7824654e7aec0ba23f16ef24e234a0bb80389fc634cc22d777d not found: ID does not exist"
Sep 16 19:18:35.651120 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.651084    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-metrics-certs\") pod \"bcca0525-447f-4048-a4ca-c5bab292d55f\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") "
Sep 16 19:18:35.651310 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.651233    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4k4w\" (UniqueName: \"kubernetes.io/projected/bcca0525-447f-4048-a4ca-c5bab292d55f-kube-api-access-l4k4w\") pod \"bcca0525-447f-4048-a4ca-c5bab292d55f\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") "
Sep 16 19:18:35.651310 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.651273    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcca0525-447f-4048-a4ca-c5bab292d55f-openshift-service-ca\") pod \"bcca0525-447f-4048-a4ca-c5bab292d55f\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") "
Sep 16 19:18:35.651430 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.651337    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-maas-api-tls\") pod \"bcca0525-447f-4048-a4ca-c5bab292d55f\" (UID: \"bcca0525-447f-4048-a4ca-c5bab292d55f\") "
Sep 16 19:18:35.651675 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.651641    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcca0525-447f-4048-a4ca-c5bab292d55f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcca0525-447f-4048-a4ca-c5bab292d55f" (UID: "bcca0525-447f-4048-a4ca-c5bab292d55f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:35.652102 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.652075    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcca0525-447f-4048-a4ca-c5bab292d55f-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:18:35.653639 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.653583    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcca0525-447f-4048-a4ca-c5bab292d55f" (UID: "bcca0525-447f-4048-a4ca-c5bab292d55f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:35.653909 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.653884    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bcca0525-447f-4048-a4ca-c5bab292d55f" (UID: "bcca0525-447f-4048-a4ca-c5bab292d55f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:35.654176 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.654146    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcca0525-447f-4048-a4ca-c5bab292d55f-kube-api-access-l4k4w" (OuterVolumeSpecName: "kube-api-access-l4k4w") pod "bcca0525-447f-4048-a4ca-c5bab292d55f" (UID: "bcca0525-447f-4048-a4ca-c5bab292d55f"). InnerVolumeSpecName "kube-api-access-l4k4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:35.753175 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.753138    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:18:35.753175 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.753170    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcca0525-447f-4048-a4ca-c5bab292d55f-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:18:35.753175 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.753181    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4k4w\" (UniqueName: \"kubernetes.io/projected/bcca0525-447f-4048-a4ca-c5bab292d55f-kube-api-access-l4k4w\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:18:35.924181 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.924137    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"]
Sep 16 19:18:35.930989 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:35.930960    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4992ef-5ff89bc5f6-c4mgk"]
Sep 16 19:18:36.620824 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:36.620790    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcca0525-447f-4048-a4ca-c5bab292d55f" path="/var/lib/kubelet/pods/bcca0525-447f-4048-a4ca-c5bab292d55f/volumes"
Sep 16 19:18:45.722383 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:18:45.722346    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:19:43.793359 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:43.793316    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"]
Sep 16 19:19:43.793919 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:43.793515    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb" podUID="77775d98-da83-4226-81ff-ea323737a5d1" containerName="maas-api" containerID="cri-o://3978a87ff838c1ac13bf05605fc1b7f10269f72386c442de5805618a1038dc35" gracePeriod=30
Sep 16 19:19:43.949225 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:43.949194    2575 generic.go:356] "Generic (PLEG): container finished" podID="77775d98-da83-4226-81ff-ea323737a5d1" containerID="3978a87ff838c1ac13bf05605fc1b7f10269f72386c442de5805618a1038dc35" exitCode=0
Sep 16 19:19:43.949386 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:43.949268    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb" event={"ID":"77775d98-da83-4226-81ff-ea323737a5d1","Type":"ContainerDied","Data":"3978a87ff838c1ac13bf05605fc1b7f10269f72386c442de5805618a1038dc35"}
Sep 16 19:19:44.054460 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.054412    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:19:44.200943 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.200905    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77775d98-da83-4226-81ff-ea323737a5d1-openshift-service-ca\") pod \"77775d98-da83-4226-81ff-ea323737a5d1\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") "
Sep 16 19:19:44.201130 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.200954    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fzdx\" (UniqueName: \"kubernetes.io/projected/77775d98-da83-4226-81ff-ea323737a5d1-kube-api-access-2fzdx\") pod \"77775d98-da83-4226-81ff-ea323737a5d1\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") "
Sep 16 19:19:44.201130 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.200997    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-maas-api-tls\") pod \"77775d98-da83-4226-81ff-ea323737a5d1\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") "
Sep 16 19:19:44.201130 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.201022    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-metrics-certs\") pod \"77775d98-da83-4226-81ff-ea323737a5d1\" (UID: \"77775d98-da83-4226-81ff-ea323737a5d1\") "
Sep 16 19:19:44.201324 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.201296    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77775d98-da83-4226-81ff-ea323737a5d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77775d98-da83-4226-81ff-ea323737a5d1" (UID: "77775d98-da83-4226-81ff-ea323737a5d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:44.203209 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.203178    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77775d98-da83-4226-81ff-ea323737a5d1" (UID: "77775d98-da83-4226-81ff-ea323737a5d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:44.203347 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.203202    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "77775d98-da83-4226-81ff-ea323737a5d1" (UID: "77775d98-da83-4226-81ff-ea323737a5d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:44.203347 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.203303    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77775d98-da83-4226-81ff-ea323737a5d1-kube-api-access-2fzdx" (OuterVolumeSpecName: "kube-api-access-2fzdx") pod "77775d98-da83-4226-81ff-ea323737a5d1" (UID: "77775d98-da83-4226-81ff-ea323737a5d1"). InnerVolumeSpecName "kube-api-access-2fzdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:44.302279 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.302245    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fzdx\" (UniqueName: \"kubernetes.io/projected/77775d98-da83-4226-81ff-ea323737a5d1-kube-api-access-2fzdx\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:19:44.302279 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.302274    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:19:44.302279 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.302283    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77775d98-da83-4226-81ff-ea323737a5d1-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:19:44.302494 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.302291    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77775d98-da83-4226-81ff-ea323737a5d1-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:19:44.955836 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.955759    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"
Sep 16 19:19:44.955836 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.955769    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb" event={"ID":"77775d98-da83-4226-81ff-ea323737a5d1","Type":"ContainerDied","Data":"bd21ddf6589d124668a0ff92b439fa6fb5309ac12f34f184d0204ffcfcc588d7"}
Sep 16 19:19:44.956394 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.955840    2575 scope.go:117] "RemoveContainer" containerID="3978a87ff838c1ac13bf05605fc1b7f10269f72386c442de5805618a1038dc35"
Sep 16 19:19:44.989041 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.989014    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"]
Sep 16 19:19:44.991321 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:44.991301    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-766919-676dd7cfb6-9t5xb"]
Sep 16 19:19:46.621933 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:46.621898    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77775d98-da83-4226-81ff-ea323737a5d1" path="/var/lib/kubelet/pods/77775d98-da83-4226-81ff-ea323737a5d1/volumes"
Sep 16 19:19:47.114968 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:19:47.114931    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:20:02.169476 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.169388    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"]
Sep 16 19:20:02.169929 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.169636    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2" podUID="d3f14aab-a37f-4766-8704-67abd2df8313" containerName="maas-api" containerID="cri-o://ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c" gracePeriod=30
Sep 16 19:20:02.233170 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.233128    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"]
Sep 16 19:20:02.233427 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.233393    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r" podUID="bab2f95b-84c8-432a-842b-60889a41cb14" containerName="payload-processing" containerID="cri-o://827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac" gracePeriod=30
Sep 16 19:20:02.353613 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.353565    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r" podUID="bab2f95b-84c8-432a-842b-60889a41cb14" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.71:9004: connect: connection refused"
Sep 16 19:20:02.353760 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.353601    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2" podUID="d3f14aab-a37f-4766-8704-67abd2df8313" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.70:8443/health\": dial tcp 10.134.0.70:8443: connect: connection refused"
Sep 16 19:20:02.515183 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.515153    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:20:02.518599 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.518576    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:20:02.582039 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.582004    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77nj9\" (UniqueName: \"kubernetes.io/projected/bab2f95b-84c8-432a-842b-60889a41cb14-kube-api-access-77nj9\") pod \"bab2f95b-84c8-432a-842b-60889a41cb14\" (UID: \"bab2f95b-84c8-432a-842b-60889a41cb14\") "
Sep 16 19:20:02.582252 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.582057    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74l4h\" (UniqueName: \"kubernetes.io/projected/d3f14aab-a37f-4766-8704-67abd2df8313-kube-api-access-74l4h\") pod \"d3f14aab-a37f-4766-8704-67abd2df8313\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") "
Sep 16 19:20:02.582252 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.582086    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-metrics-certs\") pod \"d3f14aab-a37f-4766-8704-67abd2df8313\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") "
Sep 16 19:20:02.582252 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.582121    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-maas-api-tls\") pod \"d3f14aab-a37f-4766-8704-67abd2df8313\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") "
Sep 16 19:20:02.582252 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.582157    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f14aab-a37f-4766-8704-67abd2df8313-openshift-service-ca\") pod \"d3f14aab-a37f-4766-8704-67abd2df8313\" (UID: \"d3f14aab-a37f-4766-8704-67abd2df8313\") "
Sep 16 19:20:02.582252 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.582191    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab2f95b-84c8-432a-842b-60889a41cb14-plugins-config-volume\") pod \"bab2f95b-84c8-432a-842b-60889a41cb14\" (UID: \"bab2f95b-84c8-432a-842b-60889a41cb14\") "
Sep 16 19:20:02.582702 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.582668    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f14aab-a37f-4766-8704-67abd2df8313-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3f14aab-a37f-4766-8704-67abd2df8313" (UID: "d3f14aab-a37f-4766-8704-67abd2df8313"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:02.582819 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.582769    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab2f95b-84c8-432a-842b-60889a41cb14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bab2f95b-84c8-432a-842b-60889a41cb14" (UID: "bab2f95b-84c8-432a-842b-60889a41cb14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:02.584432 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.584402    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3f14aab-a37f-4766-8704-67abd2df8313" (UID: "d3f14aab-a37f-4766-8704-67abd2df8313"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:02.584802 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.584778    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab2f95b-84c8-432a-842b-60889a41cb14-kube-api-access-77nj9" (OuterVolumeSpecName: "kube-api-access-77nj9") pod "bab2f95b-84c8-432a-842b-60889a41cb14" (UID: "bab2f95b-84c8-432a-842b-60889a41cb14"). InnerVolumeSpecName "kube-api-access-77nj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:02.584922 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.584827    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d3f14aab-a37f-4766-8704-67abd2df8313" (UID: "d3f14aab-a37f-4766-8704-67abd2df8313"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:02.584922 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.584838    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f14aab-a37f-4766-8704-67abd2df8313-kube-api-access-74l4h" (OuterVolumeSpecName: "kube-api-access-74l4h") pod "d3f14aab-a37f-4766-8704-67abd2df8313" (UID: "d3f14aab-a37f-4766-8704-67abd2df8313"). InnerVolumeSpecName "kube-api-access-74l4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:02.683056 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.682964    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab2f95b-84c8-432a-842b-60889a41cb14-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:02.683056 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.682995    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77nj9\" (UniqueName: \"kubernetes.io/projected/bab2f95b-84c8-432a-842b-60889a41cb14-kube-api-access-77nj9\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:02.683056 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.683005    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74l4h\" (UniqueName: \"kubernetes.io/projected/d3f14aab-a37f-4766-8704-67abd2df8313-kube-api-access-74l4h\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:02.683056 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.683014    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:02.683056 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.683023    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f14aab-a37f-4766-8704-67abd2df8313-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:02.683056 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:02.683032    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f14aab-a37f-4766-8704-67abd2df8313-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:03.053866 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.053764    2575 generic.go:356] "Generic (PLEG): container finished" podID="bab2f95b-84c8-432a-842b-60889a41cb14" containerID="827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac" exitCode=0
Sep 16 19:20:03.053866 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.053820    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"
Sep 16 19:20:03.053866 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.053846    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r" event={"ID":"bab2f95b-84c8-432a-842b-60889a41cb14","Type":"ContainerDied","Data":"827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac"}
Sep 16 19:20:03.054191 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.053899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r" event={"ID":"bab2f95b-84c8-432a-842b-60889a41cb14","Type":"ContainerDied","Data":"0e6792d260e215af9cc4d81418e99d46768febc3fb95ee735a5dc50622764fba"}
Sep 16 19:20:03.054191 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.053923    2575 scope.go:117] "RemoveContainer" containerID="827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac"
Sep 16 19:20:03.055194 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.055171    2575 generic.go:356] "Generic (PLEG): container finished" podID="d3f14aab-a37f-4766-8704-67abd2df8313" containerID="ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c" exitCode=0
Sep 16 19:20:03.055271 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.055219    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"
Sep 16 19:20:03.055271 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.055234    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2" event={"ID":"d3f14aab-a37f-4766-8704-67abd2df8313","Type":"ContainerDied","Data":"ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c"}
Sep 16 19:20:03.055368 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.055288    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2" event={"ID":"d3f14aab-a37f-4766-8704-67abd2df8313","Type":"ContainerDied","Data":"172d97d734add090543e304f2c514ae0121db0de0c6eb3750f52d94705bce207"}
Sep 16 19:20:03.063543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.063518    2575 scope.go:117] "RemoveContainer" containerID="827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac"
Sep 16 19:20:03.063906 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:20:03.063882    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac\": container with ID starting with 827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac not found: ID does not exist" containerID="827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac"
Sep 16 19:20:03.063982 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.063915    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac"} err="failed to get container status \"827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac\": rpc error: code = NotFound desc = could not find container \"827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac\": container with ID starting with 827c73cf0da566f2bd99355ef55eccd528d0ffb2b43e2aa4816cee60382b81ac not found: ID does not exist"
Sep 16 19:20:03.063982 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.063934    2575 scope.go:117] "RemoveContainer" containerID="ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c"
Sep 16 19:20:03.075299 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.075277    2575 scope.go:117] "RemoveContainer" containerID="ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c"
Sep 16 19:20:03.075689 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:20:03.075661    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c\": container with ID starting with ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c not found: ID does not exist" containerID="ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c"
Sep 16 19:20:03.075838 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.075701    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c"} err="failed to get container status \"ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c\": rpc error: code = NotFound desc = could not find container \"ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c\": container with ID starting with ab75bc399da6fc4297eef3cf8f5d06c09bc54e11a208a6e830bc3c4418f2148c not found: ID does not exist"
Sep 16 19:20:03.076766 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.076692    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"]
Sep 16 19:20:03.078927 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.078901    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ca3f9a-f7bb7c5b5-hnf7r"]
Sep 16 19:20:03.090539 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.090504    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"]
Sep 16 19:20:03.093500 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:03.093473    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca3f9a-67765f6449-4x9g2"]
Sep 16 19:20:04.621459 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:04.621421    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bab2f95b-84c8-432a-842b-60889a41cb14" path="/var/lib/kubelet/pods/bab2f95b-84c8-432a-842b-60889a41cb14/volumes"
Sep 16 19:20:04.621977 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:04.621955    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3f14aab-a37f-4766-8704-67abd2df8313" path="/var/lib/kubelet/pods/d3f14aab-a37f-4766-8704-67abd2df8313/volumes"
Sep 16 19:20:05.119672 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:05.119632    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:20:20.962064 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:20.962030    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"]
Sep 16 19:20:20.962545 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:20.962283    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9" podUID="8f63f930-c84d-4c94-b814-9443d2e4d07e" containerName="maas-api" containerID="cri-o://5882cefb90d62e45278a9534ac0fbb9c368461079ecb4c4dc138baa899e2cb6a" gracePeriod=30
Sep 16 19:20:21.128585 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.128547    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"]
Sep 16 19:20:21.151837 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.151796    2575 generic.go:356] "Generic (PLEG): container finished" podID="8f63f930-c84d-4c94-b814-9443d2e4d07e" containerID="5882cefb90d62e45278a9534ac0fbb9c368461079ecb4c4dc138baa899e2cb6a" exitCode=0
Sep 16 19:20:21.152099 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.151887    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9" event={"ID":"8f63f930-c84d-4c94-b814-9443d2e4d07e","Type":"ContainerDied","Data":"5882cefb90d62e45278a9534ac0fbb9c368461079ecb4c4dc138baa899e2cb6a"}
Sep 16 19:20:21.152235 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.152203    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd" podUID="2e272518-f9af-4a70-98f9-3d7d474edd13" containerName="payload-processing" containerID="cri-o://3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162" gracePeriod=30
Sep 16 19:20:21.282275 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.282250    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:20:21.362995 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.362960    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-metrics-certs\") pod \"8f63f930-c84d-4c94-b814-9443d2e4d07e\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") "
Sep 16 19:20:21.363156 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.363011    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq4v5\" (UniqueName: \"kubernetes.io/projected/8f63f930-c84d-4c94-b814-9443d2e4d07e-kube-api-access-tq4v5\") pod \"8f63f930-c84d-4c94-b814-9443d2e4d07e\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") "
Sep 16 19:20:21.363156 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.363141    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f63f930-c84d-4c94-b814-9443d2e4d07e-openshift-service-ca\") pod \"8f63f930-c84d-4c94-b814-9443d2e4d07e\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") "
Sep 16 19:20:21.363301 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.363168    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-maas-api-tls\") pod \"8f63f930-c84d-4c94-b814-9443d2e4d07e\" (UID: \"8f63f930-c84d-4c94-b814-9443d2e4d07e\") "
Sep 16 19:20:21.364034 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.363943    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f63f930-c84d-4c94-b814-9443d2e4d07e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f63f930-c84d-4c94-b814-9443d2e4d07e" (UID: "8f63f930-c84d-4c94-b814-9443d2e4d07e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:21.365408 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.365381    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f63f930-c84d-4c94-b814-9443d2e4d07e-kube-api-access-tq4v5" (OuterVolumeSpecName: "kube-api-access-tq4v5") pod "8f63f930-c84d-4c94-b814-9443d2e4d07e" (UID: "8f63f930-c84d-4c94-b814-9443d2e4d07e"). InnerVolumeSpecName "kube-api-access-tq4v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:21.365557 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.365541    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8f63f930-c84d-4c94-b814-9443d2e4d07e" (UID: "8f63f930-c84d-4c94-b814-9443d2e4d07e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:21.365646 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.365634    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f63f930-c84d-4c94-b814-9443d2e4d07e" (UID: "8f63f930-c84d-4c94-b814-9443d2e4d07e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:21.412241 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.412212    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:20:21.464004 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.463971    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f63f930-c84d-4c94-b814-9443d2e4d07e-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:21.464004 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.464002    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:21.464004 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.464014    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f63f930-c84d-4c94-b814-9443d2e4d07e-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:21.464253 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.464025    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4v5\" (UniqueName: \"kubernetes.io/projected/8f63f930-c84d-4c94-b814-9443d2e4d07e-kube-api-access-tq4v5\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:21.565315 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.565212    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqdrm\" (UniqueName: \"kubernetes.io/projected/2e272518-f9af-4a70-98f9-3d7d474edd13-kube-api-access-cqdrm\") pod \"2e272518-f9af-4a70-98f9-3d7d474edd13\" (UID: \"2e272518-f9af-4a70-98f9-3d7d474edd13\") "
Sep 16 19:20:21.565493 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.565328    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e272518-f9af-4a70-98f9-3d7d474edd13-plugins-config-volume\") pod \"2e272518-f9af-4a70-98f9-3d7d474edd13\" (UID: \"2e272518-f9af-4a70-98f9-3d7d474edd13\") "
Sep 16 19:20:21.565742 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.565690    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e272518-f9af-4a70-98f9-3d7d474edd13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e272518-f9af-4a70-98f9-3d7d474edd13" (UID: "2e272518-f9af-4a70-98f9-3d7d474edd13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:21.567382 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.567360    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e272518-f9af-4a70-98f9-3d7d474edd13-kube-api-access-cqdrm" (OuterVolumeSpecName: "kube-api-access-cqdrm") pod "2e272518-f9af-4a70-98f9-3d7d474edd13" (UID: "2e272518-f9af-4a70-98f9-3d7d474edd13"). InnerVolumeSpecName "kube-api-access-cqdrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:21.666137 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.666092    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e272518-f9af-4a70-98f9-3d7d474edd13-plugins-config-volume\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:21.666137 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:21.666128    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqdrm\" (UniqueName: \"kubernetes.io/projected/2e272518-f9af-4a70-98f9-3d7d474edd13-kube-api-access-cqdrm\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:22.158580 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.158545    2575 generic.go:356] "Generic (PLEG): container finished" podID="2e272518-f9af-4a70-98f9-3d7d474edd13" containerID="3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162" exitCode=0
Sep 16 19:20:22.159022 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.158590    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"
Sep 16 19:20:22.159022 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.158633    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd" event={"ID":"2e272518-f9af-4a70-98f9-3d7d474edd13","Type":"ContainerDied","Data":"3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162"}
Sep 16 19:20:22.159022 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.158689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd" event={"ID":"2e272518-f9af-4a70-98f9-3d7d474edd13","Type":"ContainerDied","Data":"471c6884e5258992888526b41b463b14278bd5731b8321f3b4c74b2dc3619202"}
Sep 16 19:20:22.159022 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.158740    2575 scope.go:117] "RemoveContainer" containerID="3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162"
Sep 16 19:20:22.160110 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.160093    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"
Sep 16 19:20:22.160171 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.160131    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9" event={"ID":"8f63f930-c84d-4c94-b814-9443d2e4d07e","Type":"ContainerDied","Data":"1f89b0bd40829790a3ee05e524fb4ce42324ecc101fd5bc842824f866b754251"}
Sep 16 19:20:22.168046 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.168030    2575 scope.go:117] "RemoveContainer" containerID="3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162"
Sep 16 19:20:22.168303 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:20:22.168286    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162\": container with ID starting with 3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162 not found: ID does not exist" containerID="3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162"
Sep 16 19:20:22.168378 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.168315    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162"} err="failed to get container status \"3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162\": rpc error: code = NotFound desc = could not find container \"3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162\": container with ID starting with 3396159cdcb496fedd2879d0db915cb28e9e9184cbb5add7fba3a79a42538162 not found: ID does not exist"
Sep 16 19:20:22.168378 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.168337    2575 scope.go:117] "RemoveContainer" containerID="5882cefb90d62e45278a9534ac0fbb9c368461079ecb4c4dc138baa899e2cb6a"
Sep 16 19:20:22.183910 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.183887    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"]
Sep 16 19:20:22.190192 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.190168    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2dc9fc-7744bcd57f-td5pd"]
Sep 16 19:20:22.203338 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.203311    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"]
Sep 16 19:20:22.208203 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.208181    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2dc9fc-8757dd464-j7vs9"]
Sep 16 19:20:22.621602 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.621569    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e272518-f9af-4a70-98f9-3d7d474edd13" path="/var/lib/kubelet/pods/2e272518-f9af-4a70-98f9-3d7d474edd13/volumes"
Sep 16 19:20:22.622034 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:22.622017    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f63f930-c84d-4c94-b814-9443d2e4d07e" path="/var/lib/kubelet/pods/8f63f930-c84d-4c94-b814-9443d2e4d07e/volumes"
Sep 16 19:20:24.153210 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:24.153171    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:20:32.508596 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.508557    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-md4ps"]
Sep 16 19:20:32.509128 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.508789    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66cf5649fd-md4ps" podUID="45b0b69f-cde8-485a-921c-41f0678f3477" containerName="manager" containerID="cri-o://c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121" gracePeriod=30
Sep 16 19:20:32.760305 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.760238    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 19:20:32.867294 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.867265    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-metrics-certs\") pod \"45b0b69f-cde8-485a-921c-41f0678f3477\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") "
Sep 16 19:20:32.867457 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.867326    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-webhook-cert\") pod \"45b0b69f-cde8-485a-921c-41f0678f3477\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") "
Sep 16 19:20:32.867457 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.867392    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/45b0b69f-cde8-485a-921c-41f0678f3477-kube-api-access-cwh2n\") pod \"45b0b69f-cde8-485a-921c-41f0678f3477\" (UID: \"45b0b69f-cde8-485a-921c-41f0678f3477\") "
Sep 16 19:20:32.869501 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.869472    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "45b0b69f-cde8-485a-921c-41f0678f3477" (UID: "45b0b69f-cde8-485a-921c-41f0678f3477"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:32.869610 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.869512    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b0b69f-cde8-485a-921c-41f0678f3477-kube-api-access-cwh2n" (OuterVolumeSpecName: "kube-api-access-cwh2n") pod "45b0b69f-cde8-485a-921c-41f0678f3477" (UID: "45b0b69f-cde8-485a-921c-41f0678f3477"). InnerVolumeSpecName "kube-api-access-cwh2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:32.869610 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.869523    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "45b0b69f-cde8-485a-921c-41f0678f3477" (UID: "45b0b69f-cde8-485a-921c-41f0678f3477"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:32.968451 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.968423    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-webhook-cert\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:32.968451 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.968449    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwh2n\" (UniqueName: \"kubernetes.io/projected/45b0b69f-cde8-485a-921c-41f0678f3477-kube-api-access-cwh2n\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:32.968451 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:32.968460    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b0b69f-cde8-485a-921c-41f0678f3477-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:20:33.006389 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:33.006347    2575 scope.go:117] "RemoveContainer" containerID="c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121"
Sep 16 19:20:33.218661 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:33.218619    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-md4ps" event={"ID":"45b0b69f-cde8-485a-921c-41f0678f3477","Type":"ContainerDied","Data":"c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121"}
Sep 16 19:20:33.218911 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:33.218667    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-md4ps"
Sep 16 19:20:33.218911 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:33.218693    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-md4ps" event={"ID":"45b0b69f-cde8-485a-921c-41f0678f3477","Type":"ContainerDied","Data":"bad492b858de5833e11d4f93a29a4c8316aa6f7b8baf7ba83e6a15cdf6199118"}
Sep 16 19:20:33.280535 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:33.280452    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-md4ps"]
Sep 16 19:20:33.299305 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:33.299274    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-md4ps"]
Sep 16 19:20:33.849399 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:33.849356    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:20:34.621027 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:20:34.620996    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b0b69f-cde8-485a-921c-41f0678f3477" path="/var/lib/kubelet/pods/45b0b69f-cde8-485a-921c-41f0678f3477/volumes"
Sep 16 19:20:35.027014 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:20:35.026928    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:20:35.027466 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:20:35.027053    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:20:40.493101 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:20:40.493062    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:20:40.898751 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:20:40.898693    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:20:50.542395 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:20:50.542354    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:20:55.929687 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:20:55.929651    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:21:00.552254 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:21:00.552216    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:21:10.600982 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:21:10.600932    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:21:10.890829 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:21:10.890724    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:21:20.650055 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:21:20.650018    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:21:25.889558 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:21:25.889464    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:21:30.700173 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:21:30.700132    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0b69f_cde8_485a_921c_41f0678f3477.slice/crio-conmon-c3096936b8bd60c28fb8ab86f98aa73bb28552d05522e20deb82672bfb441121.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:21:34.156802 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.156771    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66cf5649fd-xq42c"]
Sep 16 19:21:34.157238 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157222    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f63f930-c84d-4c94-b814-9443d2e4d07e" containerName="maas-api"
Sep 16 19:21:34.157281 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157240    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f63f930-c84d-4c94-b814-9443d2e4d07e" containerName="maas-api"
Sep 16 19:21:34.157281 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157253    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcca0525-447f-4048-a4ca-c5bab292d55f" containerName="maas-api"
Sep 16 19:21:34.157281 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157260    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcca0525-447f-4048-a4ca-c5bab292d55f" containerName="maas-api"
Sep 16 19:21:34.157281 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157270    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab2f95b-84c8-432a-842b-60889a41cb14" containerName="payload-processing"
Sep 16 19:21:34.157281 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157276    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab2f95b-84c8-432a-842b-60889a41cb14" containerName="payload-processing"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157287    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" containerName="maas-api"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157294    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" containerName="maas-api"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157305    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77775d98-da83-4226-81ff-ea323737a5d1" containerName="maas-api"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157310    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="77775d98-da83-4226-81ff-ea323737a5d1" containerName="maas-api"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157318    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e272518-f9af-4a70-98f9-3d7d474edd13" containerName="payload-processing"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157325    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e272518-f9af-4a70-98f9-3d7d474edd13" containerName="payload-processing"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157343    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b0b69f-cde8-485a-921c-41f0678f3477" containerName="manager"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157349    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b0b69f-cde8-485a-921c-41f0678f3477" containerName="manager"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157355    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f14aab-a37f-4766-8704-67abd2df8313" containerName="maas-api"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157360    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f14aab-a37f-4766-8704-67abd2df8313" containerName="maas-api"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157418    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e272518-f9af-4a70-98f9-3d7d474edd13" containerName="payload-processing"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157426    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3f14aab-a37f-4766-8704-67abd2df8313" containerName="maas-api"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157433    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcca0525-447f-4048-a4ca-c5bab292d55f" containerName="maas-api"
Sep 16 19:21:34.157438 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157439    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b0b69f-cde8-485a-921c-41f0678f3477" containerName="manager"
Sep 16 19:21:34.157889 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157445    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="77775d98-da83-4226-81ff-ea323737a5d1" containerName="maas-api"
Sep 16 19:21:34.157889 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157453    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bab2f95b-84c8-432a-842b-60889a41cb14" containerName="payload-processing"
Sep 16 19:21:34.157889 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157459    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f63f930-c84d-4c94-b814-9443d2e4d07e" containerName="maas-api"
Sep 16 19:21:34.157889 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.157465    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d84f0a-c5d9-4abe-bbc2-85d7fd5e8983" containerName="maas-api"
Sep 16 19:21:34.160822 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.160800    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.167580 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.167559    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 19:21:34.167700 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.167599    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zgghk\""
Sep 16 19:21:34.167700 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.167617    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 19:21:34.167700 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.167633    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 19:21:34.230245 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.230213    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-xq42c"]
Sep 16 19:21:34.235467 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.235434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-metrics-certs\") pod \"maas-controller-66cf5649fd-xq42c\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") " pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.235625 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.235566    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-kube-api-access-hc2fz\") pod \"maas-controller-66cf5649fd-xq42c\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") " pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.235625 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.235598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-webhook-cert\") pod \"maas-controller-66cf5649fd-xq42c\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") " pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.337074 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.337024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-metrics-certs\") pod \"maas-controller-66cf5649fd-xq42c\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") " pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.337260 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.337135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-kube-api-access-hc2fz\") pod \"maas-controller-66cf5649fd-xq42c\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") " pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.337260 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.337160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-webhook-cert\") pod \"maas-controller-66cf5649fd-xq42c\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") " pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.345613 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.345472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-webhook-cert\") pod \"maas-controller-66cf5649fd-xq42c\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") " pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.345613 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.345567    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-metrics-certs\") pod \"maas-controller-66cf5649fd-xq42c\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") " pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.348301 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.348276    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-kube-api-access-hc2fz\") pod \"maas-controller-66cf5649fd-xq42c\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") " pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.470876 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.470791    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:34.761110 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.761027    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-xq42c"]
Sep 16 19:21:34.764755 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:21:34.764684    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d067bd0_043d_4e3f_8f64_86f49cbf5d1a.slice/crio-f74bab22e5dd2cc463eaa2311d2ca46d63d696606fb97b84a98c6e291b4be592 WatchSource:0}: Error finding container f74bab22e5dd2cc463eaa2311d2ca46d63d696606fb97b84a98c6e291b4be592: Status 404 returned error can't find the container with id f74bab22e5dd2cc463eaa2311d2ca46d63d696606fb97b84a98c6e291b4be592
Sep 16 19:21:34.766532 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:34.766514    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:21:35.520121 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:35.520077    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-xq42c" event={"ID":"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a","Type":"ContainerStarted","Data":"bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735"}
Sep 16 19:21:35.520515 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:35.520143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-xq42c" event={"ID":"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a","Type":"ContainerStarted","Data":"f74bab22e5dd2cc463eaa2311d2ca46d63d696606fb97b84a98c6e291b4be592"}
Sep 16 19:21:35.520515 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:35.520247    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:46.527027 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:46.526993    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:46.554623 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:46.554570    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66cf5649fd-xq42c" podStartSLOduration=12.034934316 podStartE2EDuration="12.554553546s" totalImagesPullingTime="519.61923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:35.547157682 +0000 UTC m=+2523.363923267" watchObservedRunningTime="2026-09-16 19:21:46.554553546 +0000 UTC m=+2534.371319129"
Sep 16 19:21:55.976453 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:55.976411    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-xq42c"]
Sep 16 19:21:55.976958 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:55.976671    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66cf5649fd-xq42c" podUID="2d067bd0-043d-4e3f-8f64-86f49cbf5d1a" containerName="manager" containerID="cri-o://bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735" gracePeriod=30
Sep 16 19:21:56.095932 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.095901    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:21:56.226021 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.225997    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:56.348484 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.348456    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-webhook-cert\") pod \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") "
Sep 16 19:21:56.348659 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.348507    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-kube-api-access-hc2fz\") pod \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") "
Sep 16 19:21:56.348659 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.348571    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-metrics-certs\") pod \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\" (UID: \"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a\") "
Sep 16 19:21:56.350721 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.350676    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2d067bd0-043d-4e3f-8f64-86f49cbf5d1a" (UID: "2d067bd0-043d-4e3f-8f64-86f49cbf5d1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:56.350858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.350725    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-kube-api-access-hc2fz" (OuterVolumeSpecName: "kube-api-access-hc2fz") pod "2d067bd0-043d-4e3f-8f64-86f49cbf5d1a" (UID: "2d067bd0-043d-4e3f-8f64-86f49cbf5d1a"). InnerVolumeSpecName "kube-api-access-hc2fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:56.350858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.350813    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2d067bd0-043d-4e3f-8f64-86f49cbf5d1a" (UID: "2d067bd0-043d-4e3f-8f64-86f49cbf5d1a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:56.450150 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.450115    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-webhook-cert\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:21:56.450150 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.450143    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2fz\" (UniqueName: \"kubernetes.io/projected/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-kube-api-access-hc2fz\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:21:56.450150 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.450152    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:21:56.625647 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.625563    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d067bd0-043d-4e3f-8f64-86f49cbf5d1a" containerID="bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735" exitCode=0
Sep 16 19:21:56.625647 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.625613    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-xq42c"
Sep 16 19:21:56.625858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.625651    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-xq42c" event={"ID":"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a","Type":"ContainerDied","Data":"bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735"}
Sep 16 19:21:56.625858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.625721    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-xq42c" event={"ID":"2d067bd0-043d-4e3f-8f64-86f49cbf5d1a","Type":"ContainerDied","Data":"f74bab22e5dd2cc463eaa2311d2ca46d63d696606fb97b84a98c6e291b4be592"}
Sep 16 19:21:56.625858 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.625752    2575 scope.go:117] "RemoveContainer" containerID="bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735"
Sep 16 19:21:56.634797 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.634778    2575 scope.go:117] "RemoveContainer" containerID="bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735"
Sep 16 19:21:56.635068 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:21:56.635049    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735\": container with ID starting with bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735 not found: ID does not exist" containerID="bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735"
Sep 16 19:21:56.635127 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.635075    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735"} err="failed to get container status \"bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735\": rpc error: code = NotFound desc = could not find container \"bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735\": container with ID starting with bb6402a66306d65a9876f0a270d39266822740c3cda951b5b135731e657c4735 not found: ID does not exist"
Sep 16 19:21:56.650092 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.650056    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-xq42c"]
Sep 16 19:21:56.655746 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:56.655696    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-xq42c"]
Sep 16 19:21:57.524482 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.524449    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66cf5649fd-b5d5t"]
Sep 16 19:21:57.524936 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.524919    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d067bd0-043d-4e3f-8f64-86f49cbf5d1a" containerName="manager"
Sep 16 19:21:57.524981 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.524938    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d067bd0-043d-4e3f-8f64-86f49cbf5d1a" containerName="manager"
Sep 16 19:21:57.525017 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.525005    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d067bd0-043d-4e3f-8f64-86f49cbf5d1a" containerName="manager"
Sep 16 19:21:57.529275 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.529258    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.533080 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.533056    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 19:21:57.533220 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.533127    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zgghk\""
Sep 16 19:21:57.534004 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.533986    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 19:21:57.534107 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.534035    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 19:21:57.548391 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.548369    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-b5d5t"]
Sep 16 19:21:57.661693 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.661656    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b87eb1f8-d301-4066-be8f-9ce045fb9903-webhook-cert\") pod \"maas-controller-66cf5649fd-b5d5t\" (UID: \"b87eb1f8-d301-4066-be8f-9ce045fb9903\") " pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.661693 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.661699    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stb47\" (UniqueName: \"kubernetes.io/projected/b87eb1f8-d301-4066-be8f-9ce045fb9903-kube-api-access-stb47\") pod \"maas-controller-66cf5649fd-b5d5t\" (UID: \"b87eb1f8-d301-4066-be8f-9ce045fb9903\") " pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.661913 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.661830    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87eb1f8-d301-4066-be8f-9ce045fb9903-metrics-certs\") pod \"maas-controller-66cf5649fd-b5d5t\" (UID: \"b87eb1f8-d301-4066-be8f-9ce045fb9903\") " pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.762494 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.762445    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87eb1f8-d301-4066-be8f-9ce045fb9903-metrics-certs\") pod \"maas-controller-66cf5649fd-b5d5t\" (UID: \"b87eb1f8-d301-4066-be8f-9ce045fb9903\") " pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.762685 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.762518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b87eb1f8-d301-4066-be8f-9ce045fb9903-webhook-cert\") pod \"maas-controller-66cf5649fd-b5d5t\" (UID: \"b87eb1f8-d301-4066-be8f-9ce045fb9903\") " pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.762685 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.762542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stb47\" (UniqueName: \"kubernetes.io/projected/b87eb1f8-d301-4066-be8f-9ce045fb9903-kube-api-access-stb47\") pod \"maas-controller-66cf5649fd-b5d5t\" (UID: \"b87eb1f8-d301-4066-be8f-9ce045fb9903\") " pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.765035 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.765011    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b87eb1f8-d301-4066-be8f-9ce045fb9903-webhook-cert\") pod \"maas-controller-66cf5649fd-b5d5t\" (UID: \"b87eb1f8-d301-4066-be8f-9ce045fb9903\") " pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.765161 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.765139    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87eb1f8-d301-4066-be8f-9ce045fb9903-metrics-certs\") pod \"maas-controller-66cf5649fd-b5d5t\" (UID: \"b87eb1f8-d301-4066-be8f-9ce045fb9903\") " pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.771003 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.770981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stb47\" (UniqueName: \"kubernetes.io/projected/b87eb1f8-d301-4066-be8f-9ce045fb9903-kube-api-access-stb47\") pod \"maas-controller-66cf5649fd-b5d5t\" (UID: \"b87eb1f8-d301-4066-be8f-9ce045fb9903\") " pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:57.839543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:57.839517    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:58.112429 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:58.112279    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66cf5649fd-b5d5t"]
Sep 16 19:21:58.115270 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:21:58.115244    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87eb1f8_d301_4066_be8f_9ce045fb9903.slice/crio-e2399df19ec79a5dd20332241baac53a0265e64f171adfe4f117a456875fac48 WatchSource:0}: Error finding container e2399df19ec79a5dd20332241baac53a0265e64f171adfe4f117a456875fac48: Status 404 returned error can't find the container with id e2399df19ec79a5dd20332241baac53a0265e64f171adfe4f117a456875fac48
Sep 16 19:21:58.620737 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:58.620671    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d067bd0-043d-4e3f-8f64-86f49cbf5d1a" path="/var/lib/kubelet/pods/2d067bd0-043d-4e3f-8f64-86f49cbf5d1a/volumes"
Sep 16 19:21:58.637750 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:58.637703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-b5d5t" event={"ID":"b87eb1f8-d301-4066-be8f-9ce045fb9903","Type":"ContainerStarted","Data":"a611941487a317b60bb1ed7f4552dd365247165a6f79ddbefe6bc6b9b027160f"}
Sep 16 19:21:58.637868 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:58.637767    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66cf5649fd-b5d5t" event={"ID":"b87eb1f8-d301-4066-be8f-9ce045fb9903","Type":"ContainerStarted","Data":"e2399df19ec79a5dd20332241baac53a0265e64f171adfe4f117a456875fac48"}
Sep 16 19:21:58.637912 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:58.637861    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:21:58.658972 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:21:58.658922    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66cf5649fd-b5d5t" podStartSLOduration=1.258834534 podStartE2EDuration="1.658906361s" totalImagesPullingTime="400.071827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:58.658002856 +0000 UTC m=+2546.474768450" watchObservedRunningTime="2026-09-16 19:21:58.658906361 +0000 UTC m=+2546.475671946"
Sep 16 19:22:09.644560 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:22:09.644531    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66cf5649fd-b5d5t"
Sep 16 19:22:21.724231 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:22:21.724194    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:22:29.827040 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:22:29.827009    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:22:37.837180 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:22:37.837145    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:22:52.721972 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:22:52.721934    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:23:03.007785 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:23:03.007685    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:24:32.228543 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:24:32.228464    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:24:38.916458 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:24:38.916423    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:24:48.328735 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:24:48.328672    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:24:58.777077 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:24:58.777040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:25:04.117232 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.117201    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"]
Sep 16 19:25:04.120787 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.120764    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.125313 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.125288    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-651efd89-kserve-self-signed-certs\""
Sep 16 19:25:04.140229 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.140200    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"]
Sep 16 19:25:04.204957 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.204904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-model-cache\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.205147 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.204979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860d7bb5-e340-47a6-8c5f-d788982df7c7-tls-certs\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.205147 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.205006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-home\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.205147 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.205025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-tmp-dir\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.205147 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.205054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66qc\" (UniqueName: \"kubernetes.io/projected/860d7bb5-e340-47a6-8c5f-d788982df7c7-kube-api-access-h66qc\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.205147 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.205126    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-dshm\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.306531 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.306490    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-model-cache\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.306779 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.306564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860d7bb5-e340-47a6-8c5f-d788982df7c7-tls-certs\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.306779 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.306590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-home\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.306779 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.306618    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-tmp-dir\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.306779 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.306646    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66qc\" (UniqueName: \"kubernetes.io/projected/860d7bb5-e340-47a6-8c5f-d788982df7c7-kube-api-access-h66qc\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.306779 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.306754    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-dshm\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.307074 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.307026    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-model-cache\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.307133 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.307067    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-tmp-dir\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.307335 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.307309    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-home\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.309107 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.309075    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-dshm\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.309296 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.309279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860d7bb5-e340-47a6-8c5f-d788982df7c7-tls-certs\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.319254 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.319229    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66qc\" (UniqueName: \"kubernetes.io/projected/860d7bb5-e340-47a6-8c5f-d788982df7c7-kube-api-access-h66qc\") pod \"e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") " pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.430157 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.430060    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:04.707338 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:04.707306    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"]
Sep 16 19:25:04.708762 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:25:04.708725    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod860d7bb5_e340_47a6_8c5f_d788982df7c7.slice/crio-ab787ff9b73d50b86cbdf72e0cb9e1b677bfd9ddc975ec06c13ac404ca503308 WatchSource:0}: Error finding container ab787ff9b73d50b86cbdf72e0cb9e1b677bfd9ddc975ec06c13ac404ca503308: Status 404 returned error can't find the container with id ab787ff9b73d50b86cbdf72e0cb9e1b677bfd9ddc975ec06c13ac404ca503308
Sep 16 19:25:05.579299 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:05.579264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr" event={"ID":"860d7bb5-e340-47a6-8c5f-d788982df7c7","Type":"ContainerStarted","Data":"c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3"}
Sep 16 19:25:05.579693 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:05.579306    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr" event={"ID":"860d7bb5-e340-47a6-8c5f-d788982df7c7","Type":"ContainerStarted","Data":"ab787ff9b73d50b86cbdf72e0cb9e1b677bfd9ddc975ec06c13ac404ca503308"}
Sep 16 19:25:05.643514 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:05.643452    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr" podStartSLOduration=1.643433028 podStartE2EDuration="1.643433028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:05.641392168 +0000 UTC m=+2733.458157753" watchObservedRunningTime="2026-09-16 19:25:05.643433028 +0000 UTC m=+2733.460198615"
Sep 16 19:25:06.323975 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:06.323938    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:25:14.430298 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:14.430264    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:14.430764 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:14.430345    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:14.442568 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:14.442542    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:14.638470 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:14.638442    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:33.003599 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:33.003559    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"]
Sep 16 19:25:35.106064 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:35.106025    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:25:43.304070 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:43.304033    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:25:48.023959 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.023890    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr" podUID="860d7bb5-e340-47a6-8c5f-d788982df7c7" containerName="main" containerID="cri-o://c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3" gracePeriod=45
Sep 16 19:25:48.277099 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.277040    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:48.323973 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.323947    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h66qc\" (UniqueName: \"kubernetes.io/projected/860d7bb5-e340-47a6-8c5f-d788982df7c7-kube-api-access-h66qc\") pod \"860d7bb5-e340-47a6-8c5f-d788982df7c7\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") "
Sep 16 19:25:48.324132 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324015    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-dshm\") pod \"860d7bb5-e340-47a6-8c5f-d788982df7c7\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") "
Sep 16 19:25:48.324132 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324060    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860d7bb5-e340-47a6-8c5f-d788982df7c7-tls-certs\") pod \"860d7bb5-e340-47a6-8c5f-d788982df7c7\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") "
Sep 16 19:25:48.324132 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324095    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-home\") pod \"860d7bb5-e340-47a6-8c5f-d788982df7c7\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") "
Sep 16 19:25:48.324294 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324134    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-tmp-dir\") pod \"860d7bb5-e340-47a6-8c5f-d788982df7c7\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") "
Sep 16 19:25:48.324294 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324166    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-model-cache\") pod \"860d7bb5-e340-47a6-8c5f-d788982df7c7\" (UID: \"860d7bb5-e340-47a6-8c5f-d788982df7c7\") "
Sep 16 19:25:48.324392 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324323    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-home" (OuterVolumeSpecName: "home") pod "860d7bb5-e340-47a6-8c5f-d788982df7c7" (UID: "860d7bb5-e340-47a6-8c5f-d788982df7c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:48.324542 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324443    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "860d7bb5-e340-47a6-8c5f-d788982df7c7" (UID: "860d7bb5-e340-47a6-8c5f-d788982df7c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:48.324542 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324501    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "860d7bb5-e340-47a6-8c5f-d788982df7c7" (UID: "860d7bb5-e340-47a6-8c5f-d788982df7c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:48.324771 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324594    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-home\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:25:48.324771 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324614    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-tmp-dir\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:25:48.324771 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.324628    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-model-cache\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:25:48.326235 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.326205    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860d7bb5-e340-47a6-8c5f-d788982df7c7-kube-api-access-h66qc" (OuterVolumeSpecName: "kube-api-access-h66qc") pod "860d7bb5-e340-47a6-8c5f-d788982df7c7" (UID: "860d7bb5-e340-47a6-8c5f-d788982df7c7"). InnerVolumeSpecName "kube-api-access-h66qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:48.326235 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.326224    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-dshm" (OuterVolumeSpecName: "dshm") pod "860d7bb5-e340-47a6-8c5f-d788982df7c7" (UID: "860d7bb5-e340-47a6-8c5f-d788982df7c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:48.326351 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.326233    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860d7bb5-e340-47a6-8c5f-d788982df7c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "860d7bb5-e340-47a6-8c5f-d788982df7c7" (UID: "860d7bb5-e340-47a6-8c5f-d788982df7c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:48.426035 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.426000    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h66qc\" (UniqueName: \"kubernetes.io/projected/860d7bb5-e340-47a6-8c5f-d788982df7c7-kube-api-access-h66qc\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:25:48.426035 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.426027    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860d7bb5-e340-47a6-8c5f-d788982df7c7-dshm\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:25:48.426035 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.426036    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860d7bb5-e340-47a6-8c5f-d788982df7c7-tls-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:25:48.796285 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.796253    2575 generic.go:356] "Generic (PLEG): container finished" podID="860d7bb5-e340-47a6-8c5f-d788982df7c7" containerID="c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3" exitCode=0
Sep 16 19:25:48.796476 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.796294    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"
Sep 16 19:25:48.796476 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.796340    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr" event={"ID":"860d7bb5-e340-47a6-8c5f-d788982df7c7","Type":"ContainerDied","Data":"c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3"}
Sep 16 19:25:48.796476 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.796386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr" event={"ID":"860d7bb5-e340-47a6-8c5f-d788982df7c7","Type":"ContainerDied","Data":"ab787ff9b73d50b86cbdf72e0cb9e1b677bfd9ddc975ec06c13ac404ca503308"}
Sep 16 19:25:48.796476 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.796412    2575 scope.go:117] "RemoveContainer" containerID="c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3"
Sep 16 19:25:48.804925 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.804907    2575 scope.go:117] "RemoveContainer" containerID="c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3"
Sep 16 19:25:48.805248 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:25:48.805214    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3\": container with ID starting with c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3 not found: ID does not exist" containerID="c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3"
Sep 16 19:25:48.805365 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.805269    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3"} err="failed to get container status \"c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3\": rpc error: code = NotFound desc = could not find container \"c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3\": container with ID starting with c92804a5c89964577fe09ff17c76ccddd3f304dd80a23cfdfe84b76b490c7dd3 not found: ID does not exist"
Sep 16 19:25:48.815638 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.815611    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"]
Sep 16 19:25:48.819295 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:48.819265    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-651efd89-kserve-76ddf7f975-228nr"]
Sep 16 19:25:50.621279 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:50.621246    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="860d7bb5-e340-47a6-8c5f-d788982df7c7" path="/var/lib/kubelet/pods/860d7bb5-e340-47a6-8c5f-d788982df7c7/volumes"
Sep 16 19:25:51.224078 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.224044    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-756969d76c-qz5np"]
Sep 16 19:25:51.224544 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.224526    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="860d7bb5-e340-47a6-8c5f-d788982df7c7" containerName="main"
Sep 16 19:25:51.224647 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.224547    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="860d7bb5-e340-47a6-8c5f-d788982df7c7" containerName="main"
Sep 16 19:25:51.224739 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.224663    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="860d7bb5-e340-47a6-8c5f-d788982df7c7" containerName="main"
Sep 16 19:25:51.229577 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.229550    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.246358 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.246324    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-756969d76c-qz5np"]
Sep 16 19:25:51.352578 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.352541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46rz2\" (UniqueName: \"kubernetes.io/projected/85459fc8-b0ca-48ee-93c9-4b6175bfb329-kube-api-access-46rz2\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.352797 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.352583    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85459fc8-b0ca-48ee-93c9-4b6175bfb329-openshift-service-ca\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.352797 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.352675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85459fc8-b0ca-48ee-93c9-4b6175bfb329-metrics-certs\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.352797 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.352786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85459fc8-b0ca-48ee-93c9-4b6175bfb329-maas-api-tls\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.453909 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.453872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85459fc8-b0ca-48ee-93c9-4b6175bfb329-maas-api-tls\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.454134 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.453983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rz2\" (UniqueName: \"kubernetes.io/projected/85459fc8-b0ca-48ee-93c9-4b6175bfb329-kube-api-access-46rz2\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.454134 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.454011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85459fc8-b0ca-48ee-93c9-4b6175bfb329-openshift-service-ca\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.454134 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.454048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85459fc8-b0ca-48ee-93c9-4b6175bfb329-metrics-certs\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.454847 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.454813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85459fc8-b0ca-48ee-93c9-4b6175bfb329-openshift-service-ca\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.456533 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.456509    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85459fc8-b0ca-48ee-93c9-4b6175bfb329-metrics-certs\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.456729 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.456689    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85459fc8-b0ca-48ee-93c9-4b6175bfb329-maas-api-tls\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.467137 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.467110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rz2\" (UniqueName: \"kubernetes.io/projected/85459fc8-b0ca-48ee-93c9-4b6175bfb329-kube-api-access-46rz2\") pod \"maas-api-756969d76c-qz5np\" (UID: \"85459fc8-b0ca-48ee-93c9-4b6175bfb329\") " pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.542751 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.542637    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:51.615369 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.615330    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:25:51.822728 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:51.822687    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-756969d76c-qz5np"]
Sep 16 19:25:51.824478 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:25:51.824451    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85459fc8_b0ca_48ee_93c9_4b6175bfb329.slice/crio-3ba053d9876aaaca3e0ced45db8c234994672dbe8d40fbf2057fc4d0624a209c WatchSource:0}: Error finding container 3ba053d9876aaaca3e0ced45db8c234994672dbe8d40fbf2057fc4d0624a209c: Status 404 returned error can't find the container with id 3ba053d9876aaaca3e0ced45db8c234994672dbe8d40fbf2057fc4d0624a209c
Sep 16 19:25:52.820231 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:52.820140    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np" event={"ID":"85459fc8-b0ca-48ee-93c9-4b6175bfb329","Type":"ContainerStarted","Data":"cf5f9eaa242374df56b46327cb49aa94584145bdc80c551fe89b5f1b81569a72"}
Sep 16 19:25:52.820231 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:52.820202    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np" event={"ID":"85459fc8-b0ca-48ee-93c9-4b6175bfb329","Type":"ContainerStarted","Data":"3ba053d9876aaaca3e0ced45db8c234994672dbe8d40fbf2057fc4d0624a209c"}
Sep 16 19:25:52.820405 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:52.820311    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:25:52.841395 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:52.841343    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np" podStartSLOduration=1.319455854 podStartE2EDuration="1.841328581s" totalImagesPullingTime="521.872727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:52.838884342 +0000 UTC m=+2780.655649932" watchObservedRunningTime="2026-09-16 19:25:52.841328581 +0000 UTC m=+2780.658094167"
Sep 16 19:25:58.710654 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:58.710560    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:25:58.830808 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:25:58.830779    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-756969d76c-qz5np"
Sep 16 19:26:05.915019 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:05.914985    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:26:06.864627 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:06.864583    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"]
Sep 16 19:26:06.864839 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:06.864814    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697" podUID="2aab65ef-c31d-47bf-9123-238b210481bd" containerName="maas-api" containerID="cri-o://67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b" gracePeriod=30
Sep 16 19:26:08.114353 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.114328    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 19:26:08.226634 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.226549    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aab65ef-c31d-47bf-9123-238b210481bd-openshift-service-ca\") pod \"2aab65ef-c31d-47bf-9123-238b210481bd\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") "
Sep 16 19:26:08.226793 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.226658    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-metrics-certs\") pod \"2aab65ef-c31d-47bf-9123-238b210481bd\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") "
Sep 16 19:26:08.226793 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.226748    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-maas-api-tls\") pod \"2aab65ef-c31d-47bf-9123-238b210481bd\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") "
Sep 16 19:26:08.226793 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.226771    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8vbt\" (UniqueName: \"kubernetes.io/projected/2aab65ef-c31d-47bf-9123-238b210481bd-kube-api-access-b8vbt\") pod \"2aab65ef-c31d-47bf-9123-238b210481bd\" (UID: \"2aab65ef-c31d-47bf-9123-238b210481bd\") "
Sep 16 19:26:08.226979 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.226948    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aab65ef-c31d-47bf-9123-238b210481bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2aab65ef-c31d-47bf-9123-238b210481bd" (UID: "2aab65ef-c31d-47bf-9123-238b210481bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:08.228797 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.228766    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2aab65ef-c31d-47bf-9123-238b210481bd" (UID: "2aab65ef-c31d-47bf-9123-238b210481bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:08.228913 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.228806    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2aab65ef-c31d-47bf-9123-238b210481bd" (UID: "2aab65ef-c31d-47bf-9123-238b210481bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:08.228913 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.228822    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aab65ef-c31d-47bf-9123-238b210481bd-kube-api-access-b8vbt" (OuterVolumeSpecName: "kube-api-access-b8vbt") pod "2aab65ef-c31d-47bf-9123-238b210481bd" (UID: "2aab65ef-c31d-47bf-9123-238b210481bd"). InnerVolumeSpecName "kube-api-access-b8vbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:08.327818 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.327775    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-metrics-certs\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:26:08.327818 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.327810    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aab65ef-c31d-47bf-9123-238b210481bd-maas-api-tls\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:26:08.327818 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.327819    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8vbt\" (UniqueName: \"kubernetes.io/projected/2aab65ef-c31d-47bf-9123-238b210481bd-kube-api-access-b8vbt\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:26:08.327818 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.327831    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aab65ef-c31d-47bf-9123-238b210481bd-openshift-service-ca\") on node \"ip-10-0-138-234.ec2.internal\" DevicePath \"\""
Sep 16 19:26:08.905026 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.904935    2575 generic.go:356] "Generic (PLEG): container finished" podID="2aab65ef-c31d-47bf-9123-238b210481bd" containerID="67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b" exitCode=0
Sep 16 19:26:08.905026 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.904980    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"
Sep 16 19:26:08.905026 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.904990    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697" event={"ID":"2aab65ef-c31d-47bf-9123-238b210481bd","Type":"ContainerDied","Data":"67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b"}
Sep 16 19:26:08.905260 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.905038    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697" event={"ID":"2aab65ef-c31d-47bf-9123-238b210481bd","Type":"ContainerDied","Data":"83b2ec7ee1409ffb26c81f91f6305a126284f14b79827ad5d60b99262c3fe08a"}
Sep 16 19:26:08.905260 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.905062    2575 scope.go:117] "RemoveContainer" containerID="67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b"
Sep 16 19:26:08.914373 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.914350    2575 scope.go:117] "RemoveContainer" containerID="67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b"
Sep 16 19:26:08.914673 ip-10-0-138-234 kubenswrapper[2575]: E0916 19:26:08.914647    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b\": container with ID starting with 67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b not found: ID does not exist" containerID="67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b"
Sep 16 19:26:08.914749 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.914686    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b"} err="failed to get container status \"67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b\": rpc error: code = NotFound desc = could not find container \"67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b\": container with ID starting with 67ef2a97bef251cb898fd67a3acc07b81c15a6c8bfe6be503539bbf08cc9072b not found: ID does not exist"
Sep 16 19:26:08.921551 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.921527    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"]
Sep 16 19:26:08.925401 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:08.925374    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b6d4d5dc-p8697"]
Sep 16 19:26:10.620992 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:10.620954    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aab65ef-c31d-47bf-9123-238b210481bd" path="/var/lib/kubelet/pods/2aab65ef-c31d-47bf-9123-238b210481bd/volumes"
Sep 16 19:26:11.728138 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:11.728098    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:26:17.903924 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:17.903884    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mjhvr"]
Sep 16 19:26:26.300161 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:26.300129    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7996r_79856847-afeb-4b2c-aaff-5b1a103b2a7f/manager/0.log"
Sep 16 19:26:26.424913 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:26.424880    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9nmll_4a3c84cb-6463-40ba-a1f0-355e9b6e132e/manager/0.log"
Sep 16 19:26:26.537222 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:26.537190    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66cf5649fd-b5d5t_b87eb1f8-d301-4066-be8f-9ce045fb9903/manager/0.log"
Sep 16 19:26:27.131047 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:27.131018    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85745479c7-zgzbf_b87e6211-2389-4142-9e81-5c03e111052c/copy-manifests/0.log"
Sep 16 19:26:27.147119 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:27.147092    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85745479c7-zgzbf_b87e6211-2389-4142-9e81-5c03e111052c/manager/0.log"
Sep 16 19:26:27.584625 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:27.584530    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-756969d76c-qz5np_85459fc8-b0ca-48ee-93c9-4b6175bfb329/maas-api/0.log"
Sep 16 19:26:28.882499 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:28.882467    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rhgjj_c30e1190-8aec-49ac-bbf7-77e51d7602ba/manager/0.log"
Sep 16 19:26:29.231826 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:29.231746    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-mjhvr_41aca1be-34b3-426b-b830-ccf47457fde8/limitador/0.log"
Sep 16 19:26:29.347051 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:29.347001    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-q4z47_a4c325d0-b372-46e0-8138-7d181cf16ab7/manager/0.log"
Sep 16 19:26:30.441444 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:30.441408    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c4c9fb988-ncv47_0673ac02-7d60-466c-9f91-bc2f1b19eafb/router/0.log"
Sep 16 19:26:30.762337 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:30.762251    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jnz85_0c57185a-5650-4090-80c3-70763c111d18/main/0.log"
Sep 16 19:26:30.981223 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:30.981186    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-4b89j_beeabfbb-beb9-4318-a210-8c9fb293b600/main/0.log"
Sep 16 19:26:40.170088 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:40.170028    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jjs7q_fd64fe4d-7ba3-4355-84b8-8df3b98703c4/global-pull-secret-syncer/0.log"
Sep 16 19:26:40.322945 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:40.322910    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nw2jb_f6c1461d-bc9d-4506-a352-f700080df9e6/konnectivity-agent/0.log"
Sep 16 19:26:40.407940 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:40.407895    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-234.ec2.internal_96dd7b7104c0456a47b41f58b7878f94/haproxy/0.log"
Sep 16 19:26:44.325583 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:44.325551    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rhgjj_c30e1190-8aec-49ac-bbf7-77e51d7602ba/manager/0.log"
Sep 16 19:26:44.459269 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:44.459228    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-mjhvr_41aca1be-34b3-426b-b830-ccf47457fde8/limitador/0.log"
Sep 16 19:26:44.550435 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:44.550348    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-q4z47_a4c325d0-b372-46e0-8138-7d181cf16ab7/manager/0.log"
Sep 16 19:26:46.202644 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.202619    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2eb99315-c9d5-47fc-a7d0-2998af266706/alertmanager/0.log"
Sep 16 19:26:46.227371 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.227344    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2eb99315-c9d5-47fc-a7d0-2998af266706/config-reloader/0.log"
Sep 16 19:26:46.249371 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.249339    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2eb99315-c9d5-47fc-a7d0-2998af266706/kube-rbac-proxy-web/0.log"
Sep 16 19:26:46.271237 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.271205    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2eb99315-c9d5-47fc-a7d0-2998af266706/kube-rbac-proxy/0.log"
Sep 16 19:26:46.297089 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.297053    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2eb99315-c9d5-47fc-a7d0-2998af266706/kube-rbac-proxy-metric/0.log"
Sep 16 19:26:46.319230 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.319203    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2eb99315-c9d5-47fc-a7d0-2998af266706/prom-label-proxy/0.log"
Sep 16 19:26:46.345536 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.345507    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2eb99315-c9d5-47fc-a7d0-2998af266706/init-config-reloader/0.log"
Sep 16 19:26:46.412376 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.412350    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-8cz97_883516cf-651a-40be-8a75-df98f5018d25/kube-state-metrics/0.log"
Sep 16 19:26:46.438021 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.437991    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-8cz97_883516cf-651a-40be-8a75-df98f5018d25/kube-rbac-proxy-main/0.log"
Sep 16 19:26:46.460983 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.460900    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-8cz97_883516cf-651a-40be-8a75-df98f5018d25/kube-rbac-proxy-self/0.log"
Sep 16 19:26:46.502856 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.502825    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-66f699457d-v8h62_da5c4ab8-cc5e-4a1b-ad79-f7df6d3b8b99/metrics-server/0.log"
Sep 16 19:26:46.529892 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.529856    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-9pdnp_a1332e3a-cd3b-432b-a49b-842d79e355b2/monitoring-plugin/0.log"
Sep 16 19:26:46.559379 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.559347    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2shq2_fceb01e6-e703-47fd-ae5b-10fdcd3597a5/node-exporter/0.log"
Sep 16 19:26:46.581805 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.581776    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2shq2_fceb01e6-e703-47fd-ae5b-10fdcd3597a5/kube-rbac-proxy/0.log"
Sep 16 19:26:46.602772 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.602741    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2shq2_fceb01e6-e703-47fd-ae5b-10fdcd3597a5/init-textfile/0.log"
Sep 16 19:26:46.882987 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.882954    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b437de83-cad4-4a4e-ab8a-41866902bab8/prometheus/0.log"
Sep 16 19:26:46.908875 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.908841    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b437de83-cad4-4a4e-ab8a-41866902bab8/config-reloader/0.log"
Sep 16 19:26:46.932407 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.932376    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b437de83-cad4-4a4e-ab8a-41866902bab8/thanos-sidecar/0.log"
Sep 16 19:26:46.955241 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.955208    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b437de83-cad4-4a4e-ab8a-41866902bab8/kube-rbac-proxy-web/0.log"
Sep 16 19:26:46.978224 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:46.978200    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b437de83-cad4-4a4e-ab8a-41866902bab8/kube-rbac-proxy/0.log"
Sep 16 19:26:47.003808 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:47.003782    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b437de83-cad4-4a4e-ab8a-41866902bab8/kube-rbac-proxy-thanos/0.log"
Sep 16 19:26:47.033045 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:47.032968    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b437de83-cad4-4a4e-ab8a-41866902bab8/init-config-reloader/0.log"
Sep 16 19:26:47.168012 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:47.167984    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-lf6kg_3c351ff9-597c-44a0-b6c7-ebfba0998ce6/prometheus-operator-admission-webhook/0.log"
Sep 16 19:26:47.309723 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:47.309622    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-775d8ff5df-n24c5_99e4d657-08f2-41f5-bef4-d1a463f56e71/thanos-query/0.log"
Sep 16 19:26:47.337374 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:47.337345    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-775d8ff5df-n24c5_99e4d657-08f2-41f5-bef4-d1a463f56e71/kube-rbac-proxy-web/0.log"
Sep 16 19:26:47.365928 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:47.365897    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-775d8ff5df-n24c5_99e4d657-08f2-41f5-bef4-d1a463f56e71/kube-rbac-proxy/0.log"
Sep 16 19:26:47.392671 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:47.392646    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-775d8ff5df-n24c5_99e4d657-08f2-41f5-bef4-d1a463f56e71/prom-label-proxy/0.log"
Sep 16 19:26:47.424556 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:47.424528    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-775d8ff5df-n24c5_99e4d657-08f2-41f5-bef4-d1a463f56e71/kube-rbac-proxy-rules/0.log"
Sep 16 19:26:47.458425 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:47.458395    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-775d8ff5df-n24c5_99e4d657-08f2-41f5-bef4-d1a463f56e71/kube-rbac-proxy-metrics/0.log"
Sep 16 19:26:50.439874 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:50.439849    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6cf59656f5-4vbd9_d53498b9-0cb2-45fb-9c1b-4e49aa13b21d/console/0.log"
Sep 16 19:26:51.026143 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.026101    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"]
Sep 16 19:26:51.026528 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.026514    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aab65ef-c31d-47bf-9123-238b210481bd" containerName="maas-api"
Sep 16 19:26:51.026582 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.026529    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aab65ef-c31d-47bf-9123-238b210481bd" containerName="maas-api"
Sep 16 19:26:51.026634 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.026592    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aab65ef-c31d-47bf-9123-238b210481bd" containerName="maas-api"
Sep 16 19:26:51.030015 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.029991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.032575 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.032548    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mrp4v\"/\"kube-root-ca.crt\""
Sep 16 19:26:51.033400 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.033377    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mrp4v\"/\"default-dockercfg-nn9pg\""
Sep 16 19:26:51.033515 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.033377    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mrp4v\"/\"openshift-service-ca.crt\""
Sep 16 19:26:51.040782 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.040751    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-lib-modules\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.040917 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.040898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-podres\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.040962 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.040951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-sys\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.041038 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.041020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-proc\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.041074 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.041055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtd2\" (UniqueName: \"kubernetes.io/projected/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-kube-api-access-rwtd2\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.041488 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.041471    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"]
Sep 16 19:26:51.141885 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.141848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-podres\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.142082 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.141902    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-sys\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.142082 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.141940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-proc\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.142082 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.141958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtd2\" (UniqueName: \"kubernetes.io/projected/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-kube-api-access-rwtd2\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.142082 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.141977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-lib-modules\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.142082 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.142041    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-proc\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.142082 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.142066    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-sys\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.142385 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.142041    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-podres\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.142385 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.142144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-lib-modules\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.151800 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.151770    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtd2\" (UniqueName: \"kubernetes.io/projected/c55a36ea-2c5c-411e-9807-7fa1d7d1a276-kube-api-access-rwtd2\") pod \"perf-node-gather-daemonset-kml8p\" (UID: \"c55a36ea-2c5c-411e-9807-7fa1d7d1a276\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.341660 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.341628    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:51.619674 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:51.619648    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"]
Sep 16 19:26:51.621429 ip-10-0-138-234 kubenswrapper[2575]: W0916 19:26:51.621401    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc55a36ea_2c5c_411e_9807_7fa1d7d1a276.slice/crio-1cce58f31a1cafc5fa4d401174c4eed05210f2a3dec21fc77df405bb0a36c619 WatchSource:0}: Error finding container 1cce58f31a1cafc5fa4d401174c4eed05210f2a3dec21fc77df405bb0a36c619: Status 404 returned error can't find the container with id 1cce58f31a1cafc5fa4d401174c4eed05210f2a3dec21fc77df405bb0a36c619
Sep 16 19:26:52.124875 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:52.124841    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p" event={"ID":"c55a36ea-2c5c-411e-9807-7fa1d7d1a276","Type":"ContainerStarted","Data":"0641c6c59675747b6d4893451031ec805f05f586c75384c54261566dd4ccab4a"}
Sep 16 19:26:52.125058 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:52.124902    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:52.125058 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:52.124924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p" event={"ID":"c55a36ea-2c5c-411e-9807-7fa1d7d1a276","Type":"ContainerStarted","Data":"1cce58f31a1cafc5fa4d401174c4eed05210f2a3dec21fc77df405bb0a36c619"}
Sep 16 19:26:52.149575 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:52.144512    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p" podStartSLOduration=1.144485273 podStartE2EDuration="1.144485273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:52.142865636 +0000 UTC m=+2839.959631222" watchObservedRunningTime="2026-09-16 19:26:52.144485273 +0000 UTC m=+2839.961250860"
Sep 16 19:26:52.406840 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:52.406760    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jvtwj_7be3ea3a-8122-408d-b74f-54c15a936f07/dns/0.log"
Sep 16 19:26:52.429136 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:52.429109    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jvtwj_7be3ea3a-8122-408d-b74f-54c15a936f07/kube-rbac-proxy/0.log"
Sep 16 19:26:52.482450 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:52.482426    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ph95x_e06d59a6-6b82-4c92-adc3-c579ab24cffd/dns-node-resolver/0.log"
Sep 16 19:26:53.254480 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:53.254451    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bvjjb_ae87ca82-6c33-4412-87f9-92ba9f2858f3/node-ca/0.log"
Sep 16 19:26:54.896580 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:54.896548    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c4c9fb988-ncv47_0673ac02-7d60-466c-9f91-bc2f1b19eafb/router/0.log"
Sep 16 19:26:55.838128 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:55.838096    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hb7dq_9bddad80-08a6-44db-9fa4-72fb049e7307/serve-healthcheck-canary/0.log"
Sep 16 19:26:56.642875 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:56.642848    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-clcb7_8857acc3-4e6e-4f99-b382-5b9f3c460f20/kube-rbac-proxy/0.log"
Sep 16 19:26:56.665241 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:56.665212    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-clcb7_8857acc3-4e6e-4f99-b382-5b9f3c460f20/exporter/0.log"
Sep 16 19:26:56.687087 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:56.687055    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-clcb7_8857acc3-4e6e-4f99-b382-5b9f3c460f20/extractor/0.log"
Sep 16 19:26:58.140274 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:58.140244    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-kml8p"
Sep 16 19:26:58.883337 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:58.883310    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7996r_79856847-afeb-4b2c-aaff-5b1a103b2a7f/manager/0.log"
Sep 16 19:26:59.159074 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:59.158987    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9nmll_4a3c84cb-6463-40ba-a1f0-355e9b6e132e/manager/0.log"
Sep 16 19:26:59.206887 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:59.206858    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66cf5649fd-b5d5t_b87eb1f8-d301-4066-be8f-9ce045fb9903/manager/0.log"
Sep 16 19:26:59.507676 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:59.507594    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85745479c7-zgzbf_b87e6211-2389-4142-9e81-5c03e111052c/manager/0.log"
Sep 16 19:26:59.526374 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:26:59.526349    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85745479c7-zgzbf_b87e6211-2389-4142-9e81-5c03e111052c/copy-manifests/0.log"
Sep 16 19:27:00.944108 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:00.944028    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f86d45999-2br8t_55ee4c05-e435-4937-a9d9-5d5effb53a35/manager/0.log"
Sep 16 19:27:01.265171 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:01.265089    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c4c9fb988-ncv47_0673ac02-7d60-466c-9f91-bc2f1b19eafb/router/0.log"
Sep 16 19:27:03.291369 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:03.291336    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c4c9fb988-ncv47_0673ac02-7d60-466c-9f91-bc2f1b19eafb/router/0.log"
Sep 16 19:27:06.482560 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:06.482528    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-pz4lm_04d89af1-bce5-45c8-a85f-af26c0215d19/migrator/0.log"
Sep 16 19:27:06.515688 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:06.515657    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-pz4lm_04d89af1-bce5-45c8-a85f-af26c0215d19/graceful-termination/0.log"
Sep 16 19:27:06.910775 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:06.910739    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-plmd7_5ecea926-c56c-4436-9403-3484377dbfed/kube-storage-version-migrator-operator/1.log"
Sep 16 19:27:06.911535 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:06.911520    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-plmd7_5ecea926-c56c-4436-9403-3484377dbfed/kube-storage-version-migrator-operator/0.log"
Sep 16 19:27:08.074265 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:08.074233    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fgmrl_6be02496-7563-49b9-90a6-adcce7c4037e/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:27:08.100996 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:08.100964    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fgmrl_6be02496-7563-49b9-90a6-adcce7c4037e/egress-router-binary-copy/0.log"
Sep 16 19:27:08.131044 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:08.131017    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fgmrl_6be02496-7563-49b9-90a6-adcce7c4037e/cni-plugins/0.log"
Sep 16 19:27:08.160120 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:08.160095    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fgmrl_6be02496-7563-49b9-90a6-adcce7c4037e/bond-cni-plugin/0.log"
Sep 16 19:27:08.187039 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:08.187016    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fgmrl_6be02496-7563-49b9-90a6-adcce7c4037e/routeoverride-cni/0.log"
Sep 16 19:27:08.212546 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:08.212518    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fgmrl_6be02496-7563-49b9-90a6-adcce7c4037e/whereabouts-cni-bincopy/0.log"
Sep 16 19:27:08.247117 ip-10-0-138-234 kubenswrapper[2575]: I0916 19:27:08.247088    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fgmrl_6be02496-7563-49b9-90a6-adcce7c4037e/whereabouts-cni/0.log"