Sep 15 09:13:55.062961 ip-10-0-140-130 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 09:13:55.474850 ip-10-0-140-130 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 09:13:55.474850 ip-10-0-140-130 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 09:13:55.474850 ip-10-0-140-130 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 09:13:55.474850 ip-10-0-140-130 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 09:13:55.474850 ip-10-0-140-130 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 09:13:55.475482 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.474926    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 09:13:55.481456 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481440    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:13:55.481456 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481455    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:13:55.481456 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481459    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481462    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481465    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481468    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481471    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481474    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481477    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481479    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481482    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481485    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481488    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481491    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481494    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481496    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481499    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481501    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481504    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481506    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481509    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481511    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:13:55.481551 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481514    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481516    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481519    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481522    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481524    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481527    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481529    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481532    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481535    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481537    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481540    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481543    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481545    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481548    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481551    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481554    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481557    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481560    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481564    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:13:55.482045 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481567    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481571    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481576    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481578    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481581    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481584    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481587    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481589    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481592    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481594    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481597    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481600    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481602    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481605    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481608    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481611    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481613    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481616    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481619    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:13:55.482507 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481624    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481626    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481629    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481631    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481634    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481636    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481639    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481642    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481644    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481647    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481649    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481652    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481655    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481660    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481662    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481665    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481679    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481682    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481684    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481687    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:13:55.482991 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481691    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481693    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481696    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481698    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481701    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481703    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481706    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481709    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481711    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481714    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481717    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481719    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481724    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481728    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481732    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481736    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481739    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481742    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.481745    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:13:55.483475 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482455    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482463    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482466    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482469    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482472    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482475    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482478    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482481    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482484    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482487    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482489    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482491    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482494    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482497    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482499    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482502    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482504    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482507    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482510    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482512    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:13:55.483947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482515    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482518    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482520    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482523    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482526    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482530    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482532    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482535    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482538    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482540    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482543    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482545    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482547    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482550    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482553    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482555    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482558    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482560    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482563    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:13:55.484450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482566    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482569    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482571    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482574    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482576    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482579    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482581    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482600    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482604    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482607    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482610    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482613    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482615    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482619    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482623    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482626    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482629    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482633    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482637    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:13:55.484928 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482639    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482642    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482644    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482647    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482650    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482652    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482655    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482657    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482660    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482663    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482665    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482683    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482685    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482688    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482691    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482694    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482697    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482699    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482702    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482704    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:13:55.485402 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482707    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482709    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482712    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482714    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482717    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482719    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482722    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482724    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482727    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482731    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482735    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482739    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482742    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482746    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482750    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482752    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482755    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482758    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482760    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:13:55.485999 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482762    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.482765    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483139    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483143    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483146    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483149    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483152    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483154    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483158    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483160    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483163    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483165    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483168    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483171    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483173    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483176    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483178    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483181    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483185    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:13:55.486459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483188    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483191    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483194    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483197    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483200    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483203    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483206    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483208    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483211    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483213    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483216    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483218    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483221    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483223    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483226    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483229    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483231    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483234    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483236    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483238    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:13:55.486952 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483241    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483243    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483246    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483251    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483254    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483258    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483262    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483264    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483267    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483270    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483273    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483275    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483278    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483281    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483283    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483286    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483288    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483291    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483293    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483296    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:13:55.487446 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483298    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483301    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483303    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483307    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483310    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483313    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483317    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483319    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483322    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483325    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483327    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483330    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483332    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483335    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483337    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483342    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483345    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483347    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483350    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483352    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:13:55.487947 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483355    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483357    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483359    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483362    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483364    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483368    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483371    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483374    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483376    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483379    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483381    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483384    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483387    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483389    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483392    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483394    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483397    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483400    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483402    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483404    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:13:55.488460 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483407    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.483409    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483479    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483520    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483527    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483534    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483539    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483544    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483550    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483554    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483557    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483560    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483564    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483567    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483570    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483573    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483576    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483581    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483584    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483587    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483590    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483593    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483596    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483599    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 09:13:55.488956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483603    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483606    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483609    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483612    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483616    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483619    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483621    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483624    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483628    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483631    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483635    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483638    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483641    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483643    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483646    2579 flags.go:64] FLAG: --enable-server="true"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483651    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483654    2579 flags.go:64] FLAG: --event-burst="100"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483658    2579 flags.go:64] FLAG: --event-qps="50"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483661    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483664    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483667    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483684    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483687    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483690    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483693    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 15 09:13:55.489543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483697    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483700    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483703    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483706    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483709    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483712    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483715    2579 flags.go:64] FLAG: --feature-gates=""
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483719    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483722    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483724    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483728    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483731    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483734    2579 flags.go:64] FLAG: --help="false"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483736    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483740    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483743    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483746    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483749    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483752    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483755    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483758    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483761    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483767    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 09:13:55.490199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483770    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483773    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483776    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483779    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483783    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483786    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483789    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483792    2579 flags.go:64] FLAG: --lock-file=""
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483795    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483799    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483802    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483807    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483810    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483813    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483816    2579 flags.go:64] FLAG: --logging-format="text"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483819    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483822    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483825    2579 flags.go:64] FLAG: --manifest-url=""
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483828    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483835    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483838    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483842    2579 flags.go:64] FLAG: --max-pods="110"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483845    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483848    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483851    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 09:13:55.490802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483854    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483856    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483859    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483862    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483869    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483872    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483877    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483880    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483883    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483888    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483891    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483895    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483897    2579 flags.go:64] FLAG: --port="10250"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483901    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483904    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c28a5b658229f9f6"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483907    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483910    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483915    2579 flags.go:64] FLAG: --register-node="true"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483918    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483921    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483924    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483927    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483930    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483934    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483937    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 09:13:55.491440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483940    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483942    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483945    2579 flags.go:64] FLAG: --runonce="false"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483948    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483951    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483954    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483957    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483960    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483963    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483966    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483969    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483972    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483975    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483979    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483982    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483985    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483988    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483991    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.483997    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484000    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484003    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484007    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484010    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484014    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484018    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 09:13:55.492078 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484021    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484024    2579 flags.go:64] FLAG: --v="2"
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484028    2579 flags.go:64] FLAG: --version="false"
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484032    2579 flags.go:64] FLAG: --vmodule=""
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484035    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.484039    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484154    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484157    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484160    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484163    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484166    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484168    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484171    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484174    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484176    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484178    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484181    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484184    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484186    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484189    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484192    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484195    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:13:55.492714 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484198    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484201    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484203    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484206    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484209    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484211    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484214    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484217    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484219    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484223    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484226    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484228    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484231    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484233    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484236    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484239    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484241    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484244    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484246    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484249    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:13:55.493427 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484251    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484254    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484256    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484259    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484261    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484264    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484267    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484269    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484272    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484274    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484278    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484281    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484283    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484286    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484290    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484294    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484298    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484302    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:13:55.494282 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484305    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484307    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484310    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484314    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484318    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484321    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484324    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484327    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484330    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484333    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484336    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484338    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484340    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484343    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484346    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484348    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484351    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484353    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484356    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:13:55.494757 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484358    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484361    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484363    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484366    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484368    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484372    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484375    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484377    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484380    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484383    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484385    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484388    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484390    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484393    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484396    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484398    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484402    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484405    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484407    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484410    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:13:55.495244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484412    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484415    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484418    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484420    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484423    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.484425    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.485306    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.494572    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.494587    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.494602    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.494611    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494687    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494694    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494697    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494700    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:13:55.495751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494704    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494707    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494710    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494713    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494716    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494719    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494722    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494725    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494728    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494732    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494736    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494739    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494742    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494745    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494748    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494751    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494754    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494757    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494760    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:13:55.496134 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494763    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494765    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494769    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494771    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494775    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494778    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494782    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494802    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494806    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494809    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494813    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494816    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494819    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494822    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494824    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494827    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494830    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494833    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494835    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494838    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:13:55.496600 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494841    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494843    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494846    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494848    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494851    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494854    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494856    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494859    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494861    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494864    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494868    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494871    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494874    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494876    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494879    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494881    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494884    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494887    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494889    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494892    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:13:55.497116 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494895    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494897    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494900    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494902    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494905    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494907    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494910    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494912    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494915    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494917    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494920    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494923    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494926    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494931    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494933    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494936    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494945    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494947    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494950    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:13:55.497649 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494953    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494957    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494961    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494964    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494967    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494969    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494973    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494976    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494979    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494982    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494985    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494987    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494990    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494993    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494995    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.494998    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:13:55.498137 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495001    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.495006    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495115    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495120    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495123    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495126    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495129    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495132    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495134    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495137    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495140    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495142    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495145    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495147    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495158    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:13:55.498541 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495161    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495164    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495166    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495169    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495172    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495175    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495178    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495180    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495183    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495186    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495188    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495191    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495193    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495196    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495199    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495201    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495204    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495206    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495209    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495211    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:13:55.498945 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495214    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495217    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495219    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495222    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495224    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495227    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495229    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495232    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495235    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495237    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495241    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495243    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495252    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495255    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495257    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495260    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495262    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495265    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495268    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:13:55.499441 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495270    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495274    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495277    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495281    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495284    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495287    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495290    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495292    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495295    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495298    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495300    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495302    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495305    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495307    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495310    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495313    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495315    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495318    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495320    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:13:55.500008 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495323    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495326    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495329    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495331    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495334    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495336    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495339    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495347    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495350    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495352    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495355    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495359    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495362    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495365    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495367    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495370    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495372    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495375    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495378    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495380    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:13:55.500459 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495385    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495389    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495392    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495395    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495397    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495400    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495403    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:55.495406    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.495411    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.495539    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.498089    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.499078    2579 server.go:1019] "Starting client certificate rotation"
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.499175    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 09:13:55.500954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.499209    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 09:13:55.502013 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.502000    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 09:13:55.503278 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.503260    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 09:13:55.511662 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.511644    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 15 09:13:55.514937 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.514924    2579 log.go:25] "Validated CRI v1 image API"
Sep 15 09:13:55.515802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.515787    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 09:13:55.520915 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.520892    2579 fs.go:135] Filesystem UUIDs: map[25981886-e9f8-4670-a27a-31deb621e6f1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8e3c5a3e-a570-4953-863b-67a448d6790f:/dev/nvme0n1p4]
Sep 15 09:13:55.520989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.520915    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 09:13:55.526498 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.526376    2579 manager.go:216] Machine: {Timestamp:2026-09-15 09:13:55.524872027 +0000 UTC m=+0.361328991 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3076822 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21f01eff74fca3e3e375499c3a357f SystemUUID:ec21f01e-ff74-fca3-e3e3-75499c3a357f BootID:4603a10e-d1fd-4cc6-a997-c31f294cc840 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:df:c6:27:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:df:c6:27:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:72:72:ba:5f:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 09:13:55.526498 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.526489    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 09:13:55.526609 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.526565    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 09:13:55.528025 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.528010    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 09:13:55.530229 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.530205    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 09:13:55.530356 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.530231    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 09:13:55.531100 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.531089    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 09:13:55.531134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.531103    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 09:13:55.531134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.531110    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 09:13:55.531134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.531122    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 09:13:55.532404 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.532395    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 09:13:55.536870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.536859    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 09:13:55.536975 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.536966    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 09:13:55.537006 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.536996    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 09:13:55.537006 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.537005    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 09:13:55.537064 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.537019    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 09:13:55.537064 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.537030    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 15 09:13:55.537064 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.537038    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 09:13:55.537565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.537543    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 09:13:55.537686 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.537577    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 09:13:55.541867 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.541848    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 09:13:55.544156 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.544142    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 09:13:55.544210 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.544173    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 09:13:55.546706 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.546665    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 09:13:55.546779 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.546711    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 09:13:55.546940 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.546927    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 09:13:55.546970 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.546949    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 09:13:55.546970 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.546955    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 09:13:55.546970 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.546961    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 09:13:55.546970 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.546970    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 09:13:55.547070 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.546976    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 09:13:55.547070 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.546989    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 09:13:55.547070 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.546994    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 09:13:55.547070 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.547002    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 09:13:55.547070 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.547007    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 09:13:55.547070 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.547016    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 09:13:55.547070 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.547021    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 09:13:55.547938 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.547924    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 09:13:55.547982 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.547941    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 09:13:55.549403 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.549389    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-130.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 09:13:55.551697 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.551685    2579 server.go:1268] "Started kubelet"
Sep 15 09:13:55.551795 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.551768    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 09:13:55.551882 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.551832    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 09:13:55.551930 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.551905    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 09:13:55.553281 ip-10-0-140-130 systemd[1]: Started Kubernetes Kubelet.
Sep 15 09:13:55.554218 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.553948    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 09:13:55.554497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.554476    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 09:13:55.557474 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.557450    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 09:13:55.561059 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.561046    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 09:13:55.561126 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.561064    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 09:13:55.561249 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.561197    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:55.562148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.561798    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 09:13:55.562148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.561812    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 09:13:55.562148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.561797    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 09:13:55.562148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.562057    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 09:13:55.562148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.562066    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 09:13:55.564446 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.564281    2579 factory.go:55] Registering systemd factory
Sep 15 09:13:55.564446 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.564303    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 15 09:13:55.564573 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.564536    2579 factory.go:153] Registering CRI-O factory
Sep 15 09:13:55.564573 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.564551    2579 factory.go:223] Registration of the crio container factory successfully
Sep 15 09:13:55.564639 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.564610    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 09:13:55.564639 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.564633    2579 factory.go:103] Registering Raw factory
Sep 15 09:13:55.564748 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.564647    2579 manager.go:1196] Started watching for new ooms in manager
Sep 15 09:13:55.565038 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.565025    2579 manager.go:349] Starting recovery of all containers
Sep 15 09:13:55.567247 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.567223    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 09:13:55.567550 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.567527    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 09:13:55.568480 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.567315    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-130.ec2.internal.18d573b20f88e9b5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-130.ec2.internal,UID:ip-10-0-140-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-130.ec2.internal,},FirstTimestamp:2026-09-15 09:13:55.551644085 +0000 UTC m=+0.388101049,LastTimestamp:2026-09-15 09:13:55.551644085 +0000 UTC m=+0.388101049,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-130.ec2.internal,}"
Sep 15 09:13:55.576000 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.575965    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c9cml"
Sep 15 09:13:55.576257 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.576241    2579 manager.go:354] Recovery completed
Sep 15 09:13:55.581266 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.581253    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:13:55.583000 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.582885    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:13:55.583095 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.583014    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:13:55.583095 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.583026    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:13:55.583604 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.583588    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 09:13:55.583604 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.583603    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 09:13:55.583604 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.583605    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:13:55.583604 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.583617    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 09:13:55.583876 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.583853    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c9cml"
Sep 15 09:13:55.584428 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.584275    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:13:55.584428 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.584304    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:13:55.584428 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.584320    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:13:55.585037 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.584977    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-130.ec2.internal.18d573b211675a85  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-130.ec2.internal,UID:ip-10-0-140-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-130.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-130.ec2.internal,},FirstTimestamp:2026-09-15 09:13:55.582999173 +0000 UTC m=+0.419456137,LastTimestamp:2026-09-15 09:13:55.582999173 +0000 UTC m=+0.419456137,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-130.ec2.internal,}"
Sep 15 09:13:55.586549 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.586538    2579 policy_none.go:49] "None policy: Start"
Sep 15 09:13:55.586610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.586553    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 09:13:55.586610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.586564    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 09:13:55.588576 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.588566    2579 policy_none.go:47] "Start"
Sep 15 09:13:55.613408 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.613394    2579 manager.go:337] "Starting Device Plugin manager"
Sep 15 09:13:55.637457 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.613435    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 09:13:55.637457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.613445    2579 server.go:87] "Starting device plugin registration server"
Sep 15 09:13:55.637457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.613724    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 09:13:55.637457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.613734    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 09:13:55.637457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.613840    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 09:13:55.637457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.613923    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 09:13:55.637457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.613931    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 09:13:55.637457 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.614408    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 09:13:55.637457 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.614446    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:55.660712 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.660691    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 09:13:55.661821 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.661802    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 09:13:55.661900 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.661858    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 09:13:55.661900 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.661882    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 09:13:55.662071 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.661954    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 09:13:55.664363 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.664338    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:13:55.714323 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.714301    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:13:55.715210 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.715193    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:13:55.715286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.715226    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:13:55.715286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.715240    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:13:55.715286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.715269    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.722505 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.722488    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.722594 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.722515    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-130.ec2.internal\": node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:55.749896 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.749824    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:55.762988 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.762966    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal"]
Sep 15 09:13:55.763042 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.763029    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:13:55.763838 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.763822    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:13:55.763920 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.763850    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:13:55.763920 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.763861    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:13:55.766353 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.766340    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:13:55.766471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.766459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.766510 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.766487    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:13:55.767028 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.767014    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:13:55.767087 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.767041    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:13:55.767087 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.767051    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:13:55.767148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.767017    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:13:55.767148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.767118    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:13:55.767148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.767130    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:13:55.769328 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.769312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.769409 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.769337    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:13:55.770006 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.769986    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:13:55.770103 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.770020    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:13:55.770103 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.770034    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:13:55.794331 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.794309    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-130.ec2.internal\" not found" node="ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.798760 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.798742    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-130.ec2.internal\" not found" node="ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.850165 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.850146    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:55.863159 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.863124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c12dd1aab63b90cc211915d100e750d9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-130.ec2.internal\" (UID: \"c12dd1aab63b90cc211915d100e750d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.863232 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.863163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/942bbc9192b1847650e6600e6d1f86f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"942bbc9192b1847650e6600e6d1f86f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.863232 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.863181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/942bbc9192b1847650e6600e6d1f86f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"942bbc9192b1847650e6600e6d1f86f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.950251 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:55.950218    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:55.963582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.963562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/942bbc9192b1847650e6600e6d1f86f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"942bbc9192b1847650e6600e6d1f86f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.963631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.963589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/942bbc9192b1847650e6600e6d1f86f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"942bbc9192b1847650e6600e6d1f86f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.963631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.963607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c12dd1aab63b90cc211915d100e750d9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-130.ec2.internal\" (UID: \"c12dd1aab63b90cc211915d100e750d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.963727 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.963659    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/942bbc9192b1847650e6600e6d1f86f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"942bbc9192b1847650e6600e6d1f86f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.963727 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.963663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/942bbc9192b1847650e6600e6d1f86f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"942bbc9192b1847650e6600e6d1f86f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:55.963727 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:55.963663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c12dd1aab63b90cc211915d100e750d9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-130.ec2.internal\" (UID: \"c12dd1aab63b90cc211915d100e750d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:56.050997 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:56.050929    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:56.097400 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:56.097371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:56.102484 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:56.102448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:56.151416 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:56.151387    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:57.258656 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:57.258624    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:57.359164 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:57.359089    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:57.459687 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:57.459658    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:57.505051 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.505025    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 09:13:57.505616 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.505178    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 09:13:57.560271 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:57.560244    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found"
Sep 15 09:13:57.567035 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.567017    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:13:57.596777 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.596743    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 09:08:56 +0000 UTC" deadline="2028-04-11 16:52:06.916518709 +0000 UTC"
Sep 15 09:13:57.596777 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.596769    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13783h38m9.319752435s"
Sep 15 09:13:57.604713 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.604698    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:13:57.667977 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.667955    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:57.675589 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.675574    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 09:13:57.677142 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.677127    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal"
Sep 15 09:13:57.688495 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.688479    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 09:13:57.871550 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.871519    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:13:57.887762 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:57.887715    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942bbc9192b1847650e6600e6d1f86f2.slice/crio-4b85de84f95bc2d437bd5b49ffbc9264168af91eb74449a653eff0e7bbb6ff36 WatchSource:0}: Error finding container 4b85de84f95bc2d437bd5b49ffbc9264168af91eb74449a653eff0e7bbb6ff36: Status 404 returned error can't find the container with id 4b85de84f95bc2d437bd5b49ffbc9264168af91eb74449a653eff0e7bbb6ff36
Sep 15 09:13:57.888072 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:57.888051    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12dd1aab63b90cc211915d100e750d9.slice/crio-b23a0f0ba763e9a415941d71a075ff01a1f28fee246d08452e632eb162d0787a WatchSource:0}: Error finding container b23a0f0ba763e9a415941d71a075ff01a1f28fee246d08452e632eb162d0787a: Status 404 returned error can't find the container with id b23a0f0ba763e9a415941d71a075ff01a1f28fee246d08452e632eb162d0787a
Sep 15 09:13:57.893557 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:57.893543    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:13:58.319163 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.319136    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:13:58.544692 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.544659    2579 apiserver.go:52] "Watching apiserver"
Sep 15 09:13:58.552389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.552360    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 09:13:58.554607 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.554578    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-2pn5m","openshift-network-operator/iptables-alerter-qlm6h","openshift-ovn-kubernetes/ovnkube-node-7v2rw","kube-system/global-pull-secret-syncer-szh7l","kube-system/konnectivity-agent-g6sck","kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv","openshift-cluster-node-tuning-operator/tuned-kx5g8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal","openshift-multus/multus-additional-cni-plugins-psth7","openshift-multus/multus-btvt9","openshift-dns/node-resolver-2vzlg","openshift-image-registry/node-ca-vjcp6","openshift-multus/network-metrics-daemon-kzmvm"]
Sep 15 09:13:58.557384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.557366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.559836 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.559813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.560658 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.560634    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 09:13:58.560757 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.560708    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 09:13:58.560817 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.560639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 09:13:58.561096 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.560947    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 09:13:58.561096 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.561055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wc8sf\""
Sep 15 09:13:58.561901 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.561880    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 09:13:58.562282 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.562266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.563467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.563449    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nc57z\""
Sep 15 09:13:58.563934 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.563593    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 09:13:58.563934 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.563660    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 09:13:58.564072 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.563950    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:13:58.564072 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.564032    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:13:58.564364 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.564328    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 09:13:58.565498 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.565479    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 09:13:58.565684 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.565667    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 09:13:58.565991 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.565974    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 09:13:58.566578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.566173    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 09:13:58.566578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.566387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bv2fs\""
Sep 15 09:13:58.566718 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.566705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:13:58.567473 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.567450    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 09:13:58.568396 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.568312    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 09:13:58.569035 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.568915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 09:13:58.569035 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.568975    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 09:13:58.569214 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.569105    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5zcnz\""
Sep 15 09:13:58.569214 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.569159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.571146 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.571126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.571772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.571755    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-42mjt\""
Sep 15 09:13:58.571990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.571954    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 09:13:58.572161 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.572108    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 09:13:58.573029 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.573009    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 09:13:58.573905 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.573415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 09:13:58.573905 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.573781    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 09:13:58.576907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.576890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k4xkq\""
Sep 15 09:13:58.577638 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.577619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.579353 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.579325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-ovn-node-metrics-cert\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.579445 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.579389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-device-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.579502 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.579457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-cnibin\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.579649 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.579631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-var-lib-kubelet\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.579864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.579849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-hostroot\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.580048 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.580034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-run-multus-certs\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.580226 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.580210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/541af035-55ac-4b26-a32f-006da0c3a072-iptables-alerter-script\") pod \"iptables-alerter-qlm6h\" (UID: \"541af035-55ac-4b26-a32f-006da0c3a072\") " pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.580446 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.580418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-kubelet\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.580688 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.580672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt9s8\" (UniqueName: \"kubernetes.io/projected/6b6c0984-533c-4e9e-a633-28abecd84a99-kube-api-access-vt9s8\") pod \"node-resolver-2vzlg\" (UID: \"6b6c0984-533c-4e9e-a633-28abecd84a99\") " pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.580964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.580924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.581136 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-var-lib-cni-multus\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.581209 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-multus-conf-dir\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.581262 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-run-netns\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.581262 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-etc-openvswitch\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.581384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:13:58.581384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6c0984-533c-4e9e-a633-28abecd84a99-tmp-dir\") pod \"node-resolver-2vzlg\" (UID: \"6b6c0984-533c-4e9e-a633-28abecd84a99\") " pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.581490 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-socket-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.581490 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-var-lib-cni-bin\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.581599 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4aefa67f-2b69-41b7-993d-778e4589253e-multus-daemon-config\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.581599 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-ovnkube-config\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.581599 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdlj\" (UniqueName: \"kubernetes.io/projected/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-kube-api-access-jzdlj\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.581749 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b725cd06-0dd2-44be-b821-1017c1c994e9-agent-certs\") pod \"konnectivity-agent-g6sck\" (UID: \"b725cd06-0dd2-44be-b821-1017c1c994e9\") " pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:13:58.581749 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b725cd06-0dd2-44be-b821-1017c1c994e9-konnectivity-ca\") pod \"konnectivity-agent-g6sck\" (UID: \"b725cd06-0dd2-44be-b821-1017c1c994e9\") " pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:13:58.581846 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm24s\" (UniqueName: \"kubernetes.io/projected/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-kube-api-access-vm24s\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.581846 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-system-cni-dir\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.581846 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-run-k8s-cni-cncf-io\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.581983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-etc-kubernetes\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.581983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-run-openvswitch\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.581983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-run-ovn\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.581983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.581968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-env-overrides\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.582170 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d78d\" (UniqueName: \"kubernetes.io/projected/26f8b6a1-afde-4839-9886-05fc1b9f1159-kube-api-access-8d78d\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:13:58.582170 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582009    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 09:13:58.582170 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-slash\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.582170 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-run-systemd\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.582377 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-log-socket\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.582377 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-ovnkube-script-lib\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.582477 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-sys-fs\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.582531 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582489    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lf54r\""
Sep 15 09:13:58.582808 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4aefa67f-2b69-41b7-993d-778e4589253e-cni-binary-copy\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.582956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.582933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-multus-socket-dir-parent\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.583109 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-474n5\" (UniqueName: \"kubernetes.io/projected/541af035-55ac-4b26-a32f-006da0c3a072-kube-api-access-474n5\") pod \"iptables-alerter-qlm6h\" (UID: \"541af035-55ac-4b26-a32f-006da0c3a072\") " pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.583209 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583191    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 09:13:58.583302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-systemd-units\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.583450 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-node-log\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.583522 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.583575 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-cni-netd\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.583622 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6b6c0984-533c-4e9e-a633-28abecd84a99-hosts-file\") pod \"node-resolver-2vzlg\" (UID: \"6b6c0984-533c-4e9e-a633-28abecd84a99\") " pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.583622 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-cni-bin\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.583720 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.583772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-registration-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.583823 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-var-lib-openvswitch\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.583823 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-multus-cni-dir\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.583934 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583850    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.583934 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-run-netns\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.583934 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlxnl\" (UniqueName: \"kubernetes.io/projected/4aefa67f-2b69-41b7-993d-778e4589253e-kube-api-access-nlxnl\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.584077 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.583947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/541af035-55ac-4b26-a32f-006da0c3a072-host-slash\") pod \"iptables-alerter-qlm6h\" (UID: \"541af035-55ac-4b26-a32f-006da0c3a072\") " pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.584077 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.584009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.585108 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.585087    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 09:13:58.587311 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.587289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:13:58.587426 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.587372    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:13:58.589215 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.589196    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 09:13:58.589710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.589433    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4qx6q\""
Sep 15 09:13:58.589710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.589436    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 09:13:58.589710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.589664    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.591830 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.591718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:58.591830 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.591772    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:13:58.592392 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.592370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 09:13:58.592562 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.592547    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 09:13:58.592726 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.592713    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p5bs7\""
Sep 15 09:13:58.592892 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.592880    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 09:13:58.600213 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.600190    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t72pc"
Sep 15 09:13:58.610285 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.610270    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t72pc"
Sep 15 09:13:58.670356 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.670302    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 09:13:58.675809 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.675766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" event={"ID":"c12dd1aab63b90cc211915d100e750d9","Type":"ContainerStarted","Data":"b23a0f0ba763e9a415941d71a075ff01a1f28fee246d08452e632eb162d0787a"}
Sep 15 09:13:58.677837 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.677814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" event={"ID":"942bbc9192b1847650e6600e6d1f86f2","Type":"ContainerStarted","Data":"4b85de84f95bc2d437bd5b49ffbc9264168af91eb74449a653eff0e7bbb6ff36"}
Sep 15 09:13:58.684218 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-run-netns\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.684218 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6c0984-533c-4e9e-a633-28abecd84a99-tmp-dir\") pod \"node-resolver-2vzlg\" (UID: \"6b6c0984-533c-4e9e-a633-28abecd84a99\") " pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.684387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-tmp\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.684387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-tuning-conf-dir\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.684387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-var-lib-cni-bin\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.684387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-run-netns\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.684387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4aefa67f-2b69-41b7-993d-778e4589253e-multus-daemon-config\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.684387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-ovnkube-config\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.684691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-var-lib-cni-bin\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.684691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdlj\" (UniqueName: \"kubernetes.io/projected/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-kube-api-access-jzdlj\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.684691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6c0984-533c-4e9e-a633-28abecd84a99-tmp-dir\") pod \"node-resolver-2vzlg\" (UID: \"6b6c0984-533c-4e9e-a633-28abecd84a99\") " pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.684842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b725cd06-0dd2-44be-b821-1017c1c994e9-agent-certs\") pod \"konnectivity-agent-g6sck\" (UID: \"b725cd06-0dd2-44be-b821-1017c1c994e9\") " pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:13:58.684842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b725cd06-0dd2-44be-b821-1017c1c994e9-konnectivity-ca\") pod \"konnectivity-agent-g6sck\" (UID: \"b725cd06-0dd2-44be-b821-1017c1c994e9\") " pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:13:58.684842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-run-k8s-cni-cncf-io\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.684842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-etc-kubernetes\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.684842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-run-openvswitch\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.684842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-run-ovn\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-slash\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-run-systemd\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-log-socket\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4aefa67f-2b69-41b7-993d-778e4589253e-multus-daemon-config\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-ovnkube-script-lib\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-sys-fs\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-run-ovn\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-ovnkube-config\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.684967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-modprobe-d\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4aefa67f-2b69-41b7-993d-778e4589253e-cni-binary-copy\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-log-socket\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-etc-kubernetes\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-multus-socket-dir-parent\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-474n5\" (UniqueName: \"kubernetes.io/projected/541af035-55ac-4b26-a32f-006da0c3a072-kube-api-access-474n5\") pod \"iptables-alerter-qlm6h\" (UID: \"541af035-55ac-4b26-a32f-006da0c3a072\") " pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-run-systemd\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-slash\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.685120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-run-k8s-cni-cncf-io\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b725cd06-0dd2-44be-b821-1017c1c994e9-konnectivity-ca\") pod \"konnectivity-agent-g6sck\" (UID: \"b725cd06-0dd2-44be-b821-1017c1c994e9\") " pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-multus-socket-dir-parent\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-sys-fs\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-systemd-units\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-host\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-systemd-units\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-tuned\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7nrj\" (UniqueName: \"kubernetes.io/projected/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-kube-api-access-t7nrj\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-run-openvswitch\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-kubernetes\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-sysctl-conf\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-var-lib-openvswitch\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-var-lib-openvswitch\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlxnl\" (UniqueName: \"kubernetes.io/projected/4aefa67f-2b69-41b7-993d-778e4589253e-kube-api-access-nlxnl\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.686074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4aefa67f-2b69-41b7-993d-778e4589253e-cni-binary-copy\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.685975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-ovnkube-script-lib\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/541af035-55ac-4b26-a32f-006da0c3a072-host-slash\") pod \"iptables-alerter-qlm6h\" (UID: \"541af035-55ac-4b26-a32f-006da0c3a072\") " pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-run\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-sys\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f65720d5-a59e-4e28-a582-c4ae0b861511-serviceca\") pod \"node-ca-vjcp6\" (UID: \"f65720d5-a59e-4e28-a582-c4ae0b861511\") " pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-cnibin\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-var-lib-kubelet\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt9s8\" (UniqueName: \"kubernetes.io/projected/6b6c0984-533c-4e9e-a633-28abecd84a99-kube-api-access-vt9s8\") pod \"node-resolver-2vzlg\" (UID: \"6b6c0984-533c-4e9e-a633-28abecd84a99\") " pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-sysconfig\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f65720d5-a59e-4e28-a582-c4ae0b861511-host\") pod \"node-ca-vjcp6\" (UID: \"f65720d5-a59e-4e28-a582-c4ae0b861511\") " pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-etc-openvswitch\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/541af035-55ac-4b26-a32f-006da0c3a072-host-slash\") pod \"iptables-alerter-qlm6h\" (UID: \"541af035-55ac-4b26-a32f-006da0c3a072\") " pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686278    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-var-lib-kubelet\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-socket-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.686954 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.686469    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-var-lib-kubelet\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68g2q\" (UniqueName: \"kubernetes.io/projected/f65720d5-a59e-4e28-a582-c4ae0b861511-kube-api-access-68g2q\") pod \"node-ca-vjcp6\" (UID: \"f65720d5-a59e-4e28-a582-c4ae0b861511\") " pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-cni-binary-copy\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.686540    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs podName:26f8b6a1-afde-4839-9886-05fc1b9f1159 nodeName:}" failed. No retries permitted until 2026-09-15 09:13:59.186521121 +0000 UTC m=+3.016313024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs") pod "network-metrics-daemon-kzmvm" (UID: "26f8b6a1-afde-4839-9886-05fc1b9f1159") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-socket-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-system-cni-dir\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-cnibin\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-env-overrides\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d78d\" (UniqueName: \"kubernetes.io/projected/26f8b6a1-afde-4839-9886-05fc1b9f1159-kube-api-access-8d78d\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-system-cni-dir\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-etc-openvswitch\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-system-cni-dir\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-cnibin\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-sysctl-d\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-systemd\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.687990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92876d96-e3e2-4b10-ad5d-067e6ea18d22-dbus\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-env-overrides\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-node-log\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.686989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-cni-netd\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-node-log\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6b6c0984-533c-4e9e-a633-28abecd84a99-hosts-file\") pod \"node-resolver-2vzlg\" (UID: \"6b6c0984-533c-4e9e-a633-28abecd84a99\") " pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-registration-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-cni-netd\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-lib-modules\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6b6c0984-533c-4e9e-a633-28abecd84a99-hosts-file\") pod \"node-resolver-2vzlg\" (UID: \"6b6c0984-533c-4e9e-a633-28abecd84a99\") " pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-cni-bin\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-registration-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-cni-bin\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-multus-cni-dir\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.688636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-run-netns\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-ovn-node-metrics-cert\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-device-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm24s\" (UniqueName: \"kubernetes.io/projected/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-kube-api-access-vm24s\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-run-netns\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-multus-cni-dir\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-hostroot\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-run-multus-certs\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/541af035-55ac-4b26-a32f-006da0c3a072-iptables-alerter-script\") pod \"iptables-alerter-qlm6h\" (UID: \"541af035-55ac-4b26-a32f-006da0c3a072\") " pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-device-dir\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-kubelet\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-host-kubelet\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.689471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkq7\" (UniqueName: \"kubernetes.io/projected/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-kube-api-access-crkq7\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-hostroot\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-run-multus-certs\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92876d96-e3e2-4b10-ad5d-067e6ea18d22-kubelet-config\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-var-lib-cni-multus\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-multus-conf-dir\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.687750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-multus-conf-dir\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.688027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/541af035-55ac-4b26-a32f-006da0c3a072-iptables-alerter-script\") pod \"iptables-alerter-qlm6h\" (UID: \"541af035-55ac-4b26-a32f-006da0c3a072\") " pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.688189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4aefa67f-2b69-41b7-993d-778e4589253e-host-var-lib-cni-multus\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.691790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-ovn-node-metrics-cert\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.695247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.691809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b725cd06-0dd2-44be-b821-1017c1c994e9-agent-certs\") pod \"konnectivity-agent-g6sck\" (UID: \"b725cd06-0dd2-44be-b821-1017c1c994e9\") " pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:13:58.696959 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.696938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt9s8\" (UniqueName: \"kubernetes.io/projected/6b6c0984-533c-4e9e-a633-28abecd84a99-kube-api-access-vt9s8\") pod \"node-resolver-2vzlg\" (UID: \"6b6c0984-533c-4e9e-a633-28abecd84a99\") " pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.697066 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.697021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-474n5\" (UniqueName: \"kubernetes.io/projected/541af035-55ac-4b26-a32f-006da0c3a072-kube-api-access-474n5\") pod \"iptables-alerter-qlm6h\" (UID: \"541af035-55ac-4b26-a32f-006da0c3a072\") " pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.697140 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.697120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdlj\" (UniqueName: \"kubernetes.io/projected/ca7020b2-19bf-49ab-986b-f63fefcd1d1a-kube-api-access-jzdlj\") pod \"ovnkube-node-7v2rw\" (UID: \"ca7020b2-19bf-49ab-986b-f63fefcd1d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.697605 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.697580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d78d\" (UniqueName: \"kubernetes.io/projected/26f8b6a1-afde-4839-9886-05fc1b9f1159-kube-api-access-8d78d\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:13:58.697811 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.697787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm24s\" (UniqueName: \"kubernetes.io/projected/bc088506-8f23-4a92-acc8-a40a6fcf1fbc-kube-api-access-vm24s\") pod \"aws-ebs-csi-driver-node-9p9fv\" (UID: \"bc088506-8f23-4a92-acc8-a40a6fcf1fbc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.699560 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.699540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlxnl\" (UniqueName: \"kubernetes.io/projected/4aefa67f-2b69-41b7-993d-778e4589253e-kube-api-access-nlxnl\") pod \"multus-btvt9\" (UID: \"4aefa67f-2b69-41b7-993d-778e4589253e\") " pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.787978 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.787947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-lib-modules\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.787986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkq7\" (UniqueName: \"kubernetes.io/projected/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-kube-api-access-crkq7\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92876d96-e3e2-4b10-ad5d-067e6ea18d22-kubelet-config\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-tmp\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-tuning-conf-dir\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-modprobe-d\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-host\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-tuned\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7nrj\" (UniqueName: \"kubernetes.io/projected/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-kube-api-access-t7nrj\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-kubernetes\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-sysctl-conf\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-run\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-sys\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f65720d5-a59e-4e28-a582-c4ae0b861511-serviceca\") pod \"node-ca-vjcp6\" (UID: \"f65720d5-a59e-4e28-a582-c4ae0b861511\") " pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-sysconfig\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f65720d5-a59e-4e28-a582-c4ae0b861511-host\") pod \"node-ca-vjcp6\" (UID: \"f65720d5-a59e-4e28-a582-c4ae0b861511\") " pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-var-lib-kubelet\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68g2q\" (UniqueName: \"kubernetes.io/projected/f65720d5-a59e-4e28-a582-c4ae0b861511-kube-api-access-68g2q\") pod \"node-ca-vjcp6\" (UID: \"f65720d5-a59e-4e28-a582-c4ae0b861511\") " pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-cni-binary-copy\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-system-cni-dir\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-cnibin\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-sysctl-d\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-systemd\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.788689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92876d96-e3e2-4b10-ad5d-067e6ea18d22-dbus\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:58.789411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92876d96-e3e2-4b10-ad5d-067e6ea18d22-dbus\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:58.789411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.788629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-lib-modules\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.789411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.789411 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.789228    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:13:58.789411 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.789288    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret podName:92876d96-e3e2-4b10-ad5d-067e6ea18d22 nodeName:}" failed. No retries permitted until 2026-09-15 09:13:59.289271403 +0000 UTC m=+3.119063298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret") pod "global-pull-secret-syncer-szh7l" (UID: "92876d96-e3e2-4b10-ad5d-067e6ea18d22") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-sysconfig\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-cnibin\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-var-lib-kubelet\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-systemd\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-sys\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f65720d5-a59e-4e28-a582-c4ae0b861511-host\") pod \"node-ca-vjcp6\" (UID: \"f65720d5-a59e-4e28-a582-c4ae0b861511\") " pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-system-cni-dir\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-host\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.789995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-sysctl-d\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.790055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-kubernetes\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.790126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-tuning-conf-dir\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.790161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-sysctl-conf\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.790212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-modprobe-d\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.790215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-run\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.790361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-cni-binary-copy\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.790470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.790435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92876d96-e3e2-4b10-ad5d-067e6ea18d22-kubelet-config\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:58.791298 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.790624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f65720d5-a59e-4e28-a582-c4ae0b861511-serviceca\") pod \"node-ca-vjcp6\" (UID: \"f65720d5-a59e-4e28-a582-c4ae0b861511\") " pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.792999 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.792973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-etc-tuned\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.794689 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.794597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-tmp\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.798473 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.797662    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:13:58.798473 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.797687    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:13:58.798473 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.797702    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x98sq for pod openshift-network-diagnostics/network-check-target-2pn5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:13:58.798473 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:58.797756    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq podName:22d672a0-f79d-40fe-95e4-641910da6e65 nodeName:}" failed. No retries permitted until 2026-09-15 09:13:59.297740063 +0000 UTC m=+3.127531959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x98sq" (UniqueName: "kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq") pod "network-check-target-2pn5m" (UID: "22d672a0-f79d-40fe-95e4-641910da6e65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:13:58.799138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.799096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkq7\" (UniqueName: \"kubernetes.io/projected/82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03-kube-api-access-crkq7\") pod \"multus-additional-cni-plugins-psth7\" (UID: \"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03\") " pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.801007 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.800985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7nrj\" (UniqueName: \"kubernetes.io/projected/1e8697b4-942f-45e6-a1fd-51aeeba1cb77-kube-api-access-t7nrj\") pod \"tuned-kx5g8\" (UID: \"1e8697b4-942f-45e6-a1fd-51aeeba1cb77\") " pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.801585 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.801564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68g2q\" (UniqueName: \"kubernetes.io/projected/f65720d5-a59e-4e28-a582-c4ae0b861511-kube-api-access-68g2q\") pod \"node-ca-vjcp6\" (UID: \"f65720d5-a59e-4e28-a582-c4ae0b861511\") " pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:58.872925 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.872846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-btvt9"
Sep 15 09:13:58.883883 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.883846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qlm6h"
Sep 15 09:13:58.896879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.896851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:13:58.904691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.904667    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:13:58.912250 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.912225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2vzlg"
Sep 15 09:13:58.919862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.919845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv"
Sep 15 09:13:58.928398 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.928380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kx5g8"
Sep 15 09:13:58.936958 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.936940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-psth7"
Sep 15 09:13:58.943461 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:58.943446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vjcp6"
Sep 15 09:13:59.190262 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:59.190233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:13:59.190427 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:59.190354    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:13:59.190427 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:59.190407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs podName:26f8b6a1-afde-4839-9886-05fc1b9f1159 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:00.190394094 +0000 UTC m=+4.020185976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs") pod "network-metrics-daemon-kzmvm" (UID: "26f8b6a1-afde-4839-9886-05fc1b9f1159") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:13:59.290859 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:59.290822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:13:59.291021 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:59.290935    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:13:59.291021 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:59.291015    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret podName:92876d96-e3e2-4b10-ad5d-067e6ea18d22 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:00.290997184 +0000 UTC m=+4.120789088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret") pod "global-pull-secret-syncer-szh7l" (UID: "92876d96-e3e2-4b10-ad5d-067e6ea18d22") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:13:59.392107 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:59.392070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:13:59.392265 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:59.392246    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:13:59.392314 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:59.392272    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:13:59.392314 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:59.392284    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x98sq for pod openshift-network-diagnostics/network-check-target-2pn5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:13:59.392412 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:13:59.392352    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq podName:22d672a0-f79d-40fe-95e4-641910da6e65 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:00.392324189 +0000 UTC m=+4.222116091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x98sq" (UniqueName: "kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq") pod "network-check-target-2pn5m" (UID: "22d672a0-f79d-40fe-95e4-641910da6e65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:13:59.614285 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:59.614203    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 09:08:58 +0000 UTC" deadline="2028-06-03 21:01:37.865529432 +0000 UTC"
Sep 15 09:13:59.614285 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:13:59.614233    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15059h47m38.251299384s"
Sep 15 09:13:59.871313 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:59.871119    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6c0984_533c_4e9e_a633_28abecd84a99.slice/crio-ac3453990409fcc70c505e26479c2046301e8bd645b57a4163aa47457e223559 WatchSource:0}: Error finding container ac3453990409fcc70c505e26479c2046301e8bd645b57a4163aa47457e223559: Status 404 returned error can't find the container with id ac3453990409fcc70c505e26479c2046301e8bd645b57a4163aa47457e223559
Sep 15 09:13:59.876395 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:59.876367    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc088506_8f23_4a92_acc8_a40a6fcf1fbc.slice/crio-65afea29a2360078ef42319a697bb92bb4b5c8223c8c9198993e62c3fcb6cea0 WatchSource:0}: Error finding container 65afea29a2360078ef42319a697bb92bb4b5c8223c8c9198993e62c3fcb6cea0: Status 404 returned error can't find the container with id 65afea29a2360078ef42319a697bb92bb4b5c8223c8c9198993e62c3fcb6cea0
Sep 15 09:13:59.877314 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:59.877293    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aefa67f_2b69_41b7_993d_778e4589253e.slice/crio-044f0c1cbc535e2fa611e7b7e7e372bc199648e2deaf0225d9268294965b64ed WatchSource:0}: Error finding container 044f0c1cbc535e2fa611e7b7e7e372bc199648e2deaf0225d9268294965b64ed: Status 404 returned error can't find the container with id 044f0c1cbc535e2fa611e7b7e7e372bc199648e2deaf0225d9268294965b64ed
Sep 15 09:13:59.880319 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:59.880297    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7020b2_19bf_49ab_986b_f63fefcd1d1a.slice/crio-b8eb91343a6c71e889bff88909483560a8525fc7808b5adbb15c94d9016bc070 WatchSource:0}: Error finding container b8eb91343a6c71e889bff88909483560a8525fc7808b5adbb15c94d9016bc070: Status 404 returned error can't find the container with id b8eb91343a6c71e889bff88909483560a8525fc7808b5adbb15c94d9016bc070
Sep 15 09:13:59.881026 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:13:59.880823    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bcc1b7_3ce1_49b3_9e35_dcaa1f9c3f03.slice/crio-07a642a32e35bc280a6748ea0ab0ded7a3db7f05bd69a1794ce22ed3af666b0b WatchSource:0}: Error finding container 07a642a32e35bc280a6748ea0ab0ded7a3db7f05bd69a1794ce22ed3af666b0b: Status 404 returned error can't find the container with id 07a642a32e35bc280a6748ea0ab0ded7a3db7f05bd69a1794ce22ed3af666b0b
Sep 15 09:14:00.198065 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.198037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:00.198186 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.198171    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:00.198232 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.198224    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs podName:26f8b6a1-afde-4839-9886-05fc1b9f1159 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:02.198211843 +0000 UTC m=+6.028003730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs") pod "network-metrics-daemon-kzmvm" (UID: "26f8b6a1-afde-4839-9886-05fc1b9f1159") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:00.300668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.298947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:00.300668 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.299084    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:00.300668 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.299145    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret podName:92876d96-e3e2-4b10-ad5d-067e6ea18d22 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:02.299128076 +0000 UTC m=+6.128919974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret") pod "global-pull-secret-syncer-szh7l" (UID: "92876d96-e3e2-4b10-ad5d-067e6ea18d22") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:00.399662 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.399586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:00.399793 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.399771    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:14:00.399852 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.399795    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:14:00.399852 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.399809    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x98sq for pod openshift-network-diagnostics/network-check-target-2pn5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:00.399956 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.399873    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq podName:22d672a0-f79d-40fe-95e4-641910da6e65 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:02.399853949 +0000 UTC m=+6.229645846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x98sq" (UniqueName: "kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq") pod "network-check-target-2pn5m" (UID: "22d672a0-f79d-40fe-95e4-641910da6e65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:00.614979 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.614708    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 09:08:58 +0000 UTC" deadline="2028-03-07 20:44:38.48170697 +0000 UTC"
Sep 15 09:14:00.614979 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.614812    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12947h30m37.866924942s"
Sep 15 09:14:00.670397 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.669481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:00.670397 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.669616    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:00.670397 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.670086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:00.670397 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.670167    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:00.670397 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.670229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:00.670397 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:00.670328    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:00.704504 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.704439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" event={"ID":"c12dd1aab63b90cc211915d100e750d9","Type":"ContainerStarted","Data":"10fb3f81ab6e8c88a16d1d88cf64b3d12b9b104753a1cb692e27791da5f3301e"}
Sep 15 09:14:00.720630 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.720569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qlm6h" event={"ID":"541af035-55ac-4b26-a32f-006da0c3a072","Type":"ContainerStarted","Data":"c2ce61a04ad9b906e2d79475e695d3021028e40efd2bb220115cc4b4f3224567"}
Sep 15 09:14:00.730140 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.730114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vjcp6" event={"ID":"f65720d5-a59e-4e28-a582-c4ae0b861511","Type":"ContainerStarted","Data":"8f67389901029b22e8101f03411783564de996c57bce5c592143c2b15576db29"}
Sep 15 09:14:00.743715 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.743688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6sck" event={"ID":"b725cd06-0dd2-44be-b821-1017c1c994e9","Type":"ContainerStarted","Data":"7428ebe05ec9bd01bb7b1fce23dede3abd3d0a70ea368ff4b543b9ae90655a0a"}
Sep 15 09:14:00.757475 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.757450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psth7" event={"ID":"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03","Type":"ContainerStarted","Data":"07a642a32e35bc280a6748ea0ab0ded7a3db7f05bd69a1794ce22ed3af666b0b"}
Sep 15 09:14:00.764427 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.764403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-btvt9" event={"ID":"4aefa67f-2b69-41b7-993d-778e4589253e","Type":"ContainerStarted","Data":"044f0c1cbc535e2fa611e7b7e7e372bc199648e2deaf0225d9268294965b64ed"}
Sep 15 09:14:00.774470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.774445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv" event={"ID":"bc088506-8f23-4a92-acc8-a40a6fcf1fbc","Type":"ContainerStarted","Data":"65afea29a2360078ef42319a697bb92bb4b5c8223c8c9198993e62c3fcb6cea0"}
Sep 15 09:14:00.783645 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.783613    2579 generic.go:356] "Generic (PLEG): container finished" podID="942bbc9192b1847650e6600e6d1f86f2" containerID="91220dec0bd900b1c05c79671a547d3d95e29fae5104a16b84da80be88dd007c" exitCode=0
Sep 15 09:14:00.784239 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.784210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" event={"ID":"942bbc9192b1847650e6600e6d1f86f2","Type":"ContainerDied","Data":"91220dec0bd900b1c05c79671a547d3d95e29fae5104a16b84da80be88dd007c"}
Sep 15 09:14:00.794533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.794507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kx5g8" event={"ID":"1e8697b4-942f-45e6-a1fd-51aeeba1cb77","Type":"ContainerStarted","Data":"44f83046cebf3037b215560320b383f3df829c886503a80e9e05cdef7e5a7c93"}
Sep 15 09:14:00.799395 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.799350    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" podStartSLOduration=3.799322126 podStartE2EDuration="3.799322126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:00.719308894 +0000 UTC m=+4.549100797" watchObservedRunningTime="2026-09-15 09:14:00.799322126 +0000 UTC m=+4.629114028"
Sep 15 09:14:00.802913 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.802887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" event={"ID":"ca7020b2-19bf-49ab-986b-f63fefcd1d1a","Type":"ContainerStarted","Data":"b8eb91343a6c71e889bff88909483560a8525fc7808b5adbb15c94d9016bc070"}
Sep 15 09:14:00.820051 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:00.820026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2vzlg" event={"ID":"6b6c0984-533c-4e9e-a633-28abecd84a99","Type":"ContainerStarted","Data":"ac3453990409fcc70c505e26479c2046301e8bd645b57a4163aa47457e223559"}
Sep 15 09:14:01.843063 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:01.842912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" event={"ID":"942bbc9192b1847650e6600e6d1f86f2","Type":"ContainerStarted","Data":"a1e03512e95df871a6bbd346b29e5d1979af6362f6cce822991f5a28bd2bef60"}
Sep 15 09:14:01.857047 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:01.856994    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" podStartSLOduration=4.856949316 podStartE2EDuration="4.856949316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:01.856425168 +0000 UTC m=+5.686217074" watchObservedRunningTime="2026-09-15 09:14:01.856949316 +0000 UTC m=+5.686741220"
Sep 15 09:14:02.215034 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:02.214996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:02.215200 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.215139    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:02.215274 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.215205    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs podName:26f8b6a1-afde-4839-9886-05fc1b9f1159 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:06.215184904 +0000 UTC m=+10.044976799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs") pod "network-metrics-daemon-kzmvm" (UID: "26f8b6a1-afde-4839-9886-05fc1b9f1159") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:02.315779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:02.315743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:02.315943 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.315932    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:02.315999 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.315990    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret podName:92876d96-e3e2-4b10-ad5d-067e6ea18d22 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:06.315973707 +0000 UTC m=+10.145765596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret") pod "global-pull-secret-syncer-szh7l" (UID: "92876d96-e3e2-4b10-ad5d-067e6ea18d22") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:02.417044 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:02.416983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:02.417252 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.417230    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:14:02.417330 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.417258    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:14:02.417330 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.417272    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x98sq for pod openshift-network-diagnostics/network-check-target-2pn5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:02.417330 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.417326    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq podName:22d672a0-f79d-40fe-95e4-641910da6e65 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:06.417310026 +0000 UTC m=+10.247101921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x98sq" (UniqueName: "kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq") pod "network-check-target-2pn5m" (UID: "22d672a0-f79d-40fe-95e4-641910da6e65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:02.672130 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:02.671244    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:02.672130 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.671382    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:02.672130 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:02.671796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:02.672130 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.671902    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:02.672130 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:02.671956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:02.672130 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:02.672041    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:04.671955 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:04.671031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:04.671955 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:04.671179    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:04.671955 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:04.671696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:04.671955 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:04.671787    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:04.671955 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:04.671934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:04.672601 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:04.672022    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:06.247737 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:06.247700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:06.248217 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.247838    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:06.248217 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.247903    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs podName:26f8b6a1-afde-4839-9886-05fc1b9f1159 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:14.24788812 +0000 UTC m=+18.077680001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs") pod "network-metrics-daemon-kzmvm" (UID: "26f8b6a1-afde-4839-9886-05fc1b9f1159") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:06.348297 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:06.348258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:06.348486 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.348415    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:06.348486 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.348483    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret podName:92876d96-e3e2-4b10-ad5d-067e6ea18d22 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:14.34846275 +0000 UTC m=+18.178254649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret") pod "global-pull-secret-syncer-szh7l" (UID: "92876d96-e3e2-4b10-ad5d-067e6ea18d22") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:06.449593 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:06.449560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:06.449794 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.449746    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:14:06.449794 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.449766    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:14:06.449794 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.449775    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x98sq for pod openshift-network-diagnostics/network-check-target-2pn5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:06.449937 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.449834    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq podName:22d672a0-f79d-40fe-95e4-641910da6e65 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:14.449816091 +0000 UTC m=+18.279607973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x98sq" (UniqueName: "kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq") pod "network-check-target-2pn5m" (UID: "22d672a0-f79d-40fe-95e4-641910da6e65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:06.670089 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:06.670059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:06.670255 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.670183    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:06.670553 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:06.670358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:06.670553 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:06.670417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:06.670553 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.670511    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:06.670748 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:06.670581    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:08.669577 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:08.669543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:08.670027 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:08.669544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:08.670027 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:08.669662    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:08.670027 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:08.669545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:08.670027 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:08.669771    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:08.670027 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:08.669793    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:10.669416 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:10.669130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:10.669812 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:10.669465    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:10.669812 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:10.669130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:10.669812 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:10.669141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:10.669812 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:10.669575    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:10.669812 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:10.669625    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:12.668976 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:12.668944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:12.669400 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:12.668944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:12.669400 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:12.669083    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:12.669400 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:12.669092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:12.669400 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:12.669181    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:12.669400 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:12.669274    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:14.304109 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:14.304076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:14.304578 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.304206    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:14.304578 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.304264    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs podName:26f8b6a1-afde-4839-9886-05fc1b9f1159 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:30.304250369 +0000 UTC m=+34.134042263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs") pod "network-metrics-daemon-kzmvm" (UID: "26f8b6a1-afde-4839-9886-05fc1b9f1159") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:14.405360 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:14.405319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:14.405516 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.405453    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:14.405558 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.405521    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret podName:92876d96-e3e2-4b10-ad5d-067e6ea18d22 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:30.405499069 +0000 UTC m=+34.235290977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret") pod "global-pull-secret-syncer-szh7l" (UID: "92876d96-e3e2-4b10-ad5d-067e6ea18d22") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:14.506137 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:14.506102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:14.506319 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.506292    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:14:14.506319 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.506322    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:14:14.506464 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.506350    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x98sq for pod openshift-network-diagnostics/network-check-target-2pn5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:14.506464 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.506417    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq podName:22d672a0-f79d-40fe-95e4-641910da6e65 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:30.50639618 +0000 UTC m=+34.336188084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x98sq" (UniqueName: "kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq") pod "network-check-target-2pn5m" (UID: "22d672a0-f79d-40fe-95e4-641910da6e65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:14.669279 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:14.669242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:14.669430 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:14.669247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:14.669430 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.669364    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:14.669430 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:14.669383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:14.669564 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.669456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:14.669564 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:14.669522    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:16.669666 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:16.669635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:16.670099 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:16.669718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:16.670099 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:16.669738    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:16.670099 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:16.669827    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:16.670099 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:16.669861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:16.670099 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:16.669931    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:17.880234 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.879668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vjcp6" event={"ID":"f65720d5-a59e-4e28-a582-c4ae0b861511","Type":"ContainerStarted","Data":"d5fa3a8f670c6f25c6761ab5e7c3d34dc855967340abdfd19cf46231ef798fb8"}
Sep 15 09:14:17.881233 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.881209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6sck" event={"ID":"b725cd06-0dd2-44be-b821-1017c1c994e9","Type":"ContainerStarted","Data":"165eccad0fc01fd49824e808ec36f8a3a9298ae63f04965088def4340670ce79"}
Sep 15 09:14:17.882751 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.882728    2579 generic.go:356] "Generic (PLEG): container finished" podID="82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03" containerID="a757ea1c2b8d69943d5bc3e74c83494f8194018b0f0e1f1a3599d64c68bc84d7" exitCode=0
Sep 15 09:14:17.882864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.882807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psth7" event={"ID":"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03","Type":"ContainerDied","Data":"a757ea1c2b8d69943d5bc3e74c83494f8194018b0f0e1f1a3599d64c68bc84d7"}
Sep 15 09:14:17.884701 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.884675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-btvt9" event={"ID":"4aefa67f-2b69-41b7-993d-778e4589253e","Type":"ContainerStarted","Data":"04430f92dc631c6a52fb7da72003801293e472b08114a5212b8c2ecb3321274a"}
Sep 15 09:14:17.886642 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.886618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv" event={"ID":"bc088506-8f23-4a92-acc8-a40a6fcf1fbc","Type":"ContainerStarted","Data":"7122e1f1a2c0d236437585cab3186fce355a31d5223b60f69b3d6580b071b707"}
Sep 15 09:14:17.888384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.888357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kx5g8" event={"ID":"1e8697b4-942f-45e6-a1fd-51aeeba1cb77","Type":"ContainerStarted","Data":"ff35edbf68da8910a0c1ada162537d5df89b43a3879c0adb59564160b3677c63"}
Sep 15 09:14:17.891481 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.891462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" event={"ID":"ca7020b2-19bf-49ab-986b-f63fefcd1d1a","Type":"ContainerStarted","Data":"3b88709ddcb9d4b9cb974caecc21061d5b4057f9e65879b79cb2dff9400274be"}
Sep 15 09:14:17.891572 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.891524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" event={"ID":"ca7020b2-19bf-49ab-986b-f63fefcd1d1a","Type":"ContainerStarted","Data":"8537ee803b7a48c6b57b937c8781460c9adb9584e7f5154cfb721027f274b522"}
Sep 15 09:14:17.891572 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.891561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" event={"ID":"ca7020b2-19bf-49ab-986b-f63fefcd1d1a","Type":"ContainerStarted","Data":"72d71e3318c2fc18478fd9ee8f9c0b85567b3d3d8eaff59b7a7b977ec4a9bb3c"}
Sep 15 09:14:17.891686 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.891603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" event={"ID":"ca7020b2-19bf-49ab-986b-f63fefcd1d1a","Type":"ContainerStarted","Data":"be5dee41e07ecf8ab7ed32b5fbbbaa601ff981b7511656dc2a4e84e8570d3cd5"}
Sep 15 09:14:17.893086 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.893067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2vzlg" event={"ID":"6b6c0984-533c-4e9e-a633-28abecd84a99","Type":"ContainerStarted","Data":"c2aaf96f28b1dbc5d8820fd1e7e84afe213d0165c650d3571657a75c52fda9c3"}
Sep 15 09:14:17.893569 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.893522    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vjcp6" podStartSLOduration=3.332332381 podStartE2EDuration="20.893505655s" totalImagesPullingTime="17.561173274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:17.892778775 +0000 UTC m=+21.722570689" watchObservedRunningTime="2026-09-15 09:14:17.893505655 +0000 UTC m=+21.723297558"
Sep 15 09:14:17.907849 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.907807    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kx5g8" podStartSLOduration=3.343134507 podStartE2EDuration="20.907795829s" totalImagesPullingTime="17.564661322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:17.907726714 +0000 UTC m=+21.737518618" watchObservedRunningTime="2026-09-15 09:14:17.907795829 +0000 UTC m=+21.737587732"
Sep 15 09:14:17.948536 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.948500    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-btvt9" podStartSLOduration=4.224560059 podStartE2EDuration="21.94848852s" totalImagesPullingTime="17.723928461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:17.945671054 +0000 UTC m=+21.775462956" watchObservedRunningTime="2026-09-15 09:14:17.94848852 +0000 UTC m=+21.778280423"
Sep 15 09:14:17.967901 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.967863    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g6sck" podStartSLOduration=8.404899505 podStartE2EDuration="21.967851344s" totalImagesPullingTime="13.562951839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:17.967681906 +0000 UTC m=+21.797473809" watchObservedRunningTime="2026-09-15 09:14:17.967851344 +0000 UTC m=+21.797643244"
Sep 15 09:14:17.982157 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:17.982124    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2vzlg" podStartSLOduration=4.407789166 podStartE2EDuration="21.982113552s" totalImagesPullingTime="17.574324386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:17.981812316 +0000 UTC m=+21.811604218" watchObservedRunningTime="2026-09-15 09:14:17.982113552 +0000 UTC m=+21.811905455"
Sep 15 09:14:18.669695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:18.669518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:18.669695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:18.669518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:18.669695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:18.669654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:18.669695 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:18.669652    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:18.670079 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:18.669752    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:18.670079 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:18.669845    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:18.897082 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:18.896991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qlm6h" event={"ID":"541af035-55ac-4b26-a32f-006da0c3a072","Type":"ContainerStarted","Data":"a8a68cfd735b8055a2ead29559fef84699a1c844b054c4c3a7df91c9a6696750"}
Sep 15 09:14:18.901865 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:18.900865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" event={"ID":"ca7020b2-19bf-49ab-986b-f63fefcd1d1a","Type":"ContainerStarted","Data":"c956b3cbb52a5e2e95ec61d3712de4e79ce28b362bb4c59c43e82eabf6a099de"}
Sep 15 09:14:18.901865 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:18.901019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" event={"ID":"ca7020b2-19bf-49ab-986b-f63fefcd1d1a","Type":"ContainerStarted","Data":"0e13f3c5aac8a28c4e3d703a7697a1f3c7dfab4fd1f789d5feda059e1c549610"}
Sep 15 09:14:19.282277 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:19.282077    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 09:14:19.635527 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:19.635440    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T09:14:19.282274779Z","UUID":"2863c3bd-baa5-498f-aa9f-ad9ec1cb228e","Handler":null,"Name":"","Endpoint":""}
Sep 15 09:14:19.638316 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:19.638292    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 09:14:19.638469 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:19.638327    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 09:14:19.906871 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:19.906836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv" event={"ID":"bc088506-8f23-4a92-acc8-a40a6fcf1fbc","Type":"ContainerStarted","Data":"7d501b9e69026a49a389021d45a0c970c1897485a022a7c8136aa45f93c986a2"}
Sep 15 09:14:20.669581 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:20.669546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:20.669728 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:20.669591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:20.669728 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:20.669550    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:20.669728 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:20.669672    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:20.669890 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:20.669782    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:20.669890 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:20.669867    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:20.913250 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:20.913220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" event={"ID":"ca7020b2-19bf-49ab-986b-f63fefcd1d1a","Type":"ContainerStarted","Data":"fcc1dc7881a5da7fe2640f41eae36d7fb57e2b17b66e63b17895897838629421"}
Sep 15 09:14:20.915561 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:20.915532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv" event={"ID":"bc088506-8f23-4a92-acc8-a40a6fcf1fbc","Type":"ContainerStarted","Data":"5b2486d3328c5bc3ab08ca61da53f82dca97c76ef45a79a5bf253c1a75cebcaa"}
Sep 15 09:14:20.933347 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:20.933296    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9p9fv" podStartSLOduration=3.682502746 podStartE2EDuration="23.93328085s" totalImagesPullingTime="20.250778104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:20.93312332 +0000 UTC m=+24.762915224" watchObservedRunningTime="2026-09-15 09:14:20.93328085 +0000 UTC m=+24.763072755"
Sep 15 09:14:20.933843 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:20.933806    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qlm6h" podStartSLOduration=7.37364576 podStartE2EDuration="24.933795313s" totalImagesPullingTime="17.560149553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:18.910729946 +0000 UTC m=+22.740521847" watchObservedRunningTime="2026-09-15 09:14:20.933795313 +0000 UTC m=+24.763587216"
Sep 15 09:14:22.310857 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.310825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:14:22.311413 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.311390    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:14:22.669517 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.669489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:22.669678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.669491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:22.669678 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:22.669611    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:22.669678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.669628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:22.669817 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:22.669732    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:22.669892 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:22.669866    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:22.925192 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.925161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" event={"ID":"ca7020b2-19bf-49ab-986b-f63fefcd1d1a","Type":"ContainerStarted","Data":"b486f6815546e03043be3072c105767e28ed95412f6c815f8d282150df4ac751"}
Sep 15 09:14:22.925954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.925564    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:14:22.925954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.925647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:14:22.925954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.925672    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:14:22.925954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.925724    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:14:22.926545 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.926519    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g6sck"
Sep 15 09:14:22.943046 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.942898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:14:22.944012 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.943995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:14:22.952447 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:22.952089    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw" podStartSLOduration=9.362088707 podStartE2EDuration="26.952071217s" totalImagesPullingTime="17.58998251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:22.950546013 +0000 UTC m=+26.780337918" watchObservedRunningTime="2026-09-15 09:14:22.952071217 +0000 UTC m=+26.781863120"
Sep 15 09:14:23.929456 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:23.929418    2579 generic.go:356] "Generic (PLEG): container finished" podID="82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03" containerID="9e6dc7ec30c3bce9013907741a421132d1023ffde757b03557a374c8ef928542" exitCode=0
Sep 15 09:14:23.930068 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:23.929509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psth7" event={"ID":"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03","Type":"ContainerDied","Data":"9e6dc7ec30c3bce9013907741a421132d1023ffde757b03557a374c8ef928542"}
Sep 15 09:14:24.669985 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:24.669948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:24.669985 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:24.669971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:24.670209 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:24.669949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:24.670209 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:24.670092    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:24.670312 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:24.670204    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:24.670396 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:24.670308    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:24.820271 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:24.820243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-szh7l"]
Sep 15 09:14:24.826504 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:24.826480    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2pn5m"]
Sep 15 09:14:24.827327 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:24.827297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kzmvm"]
Sep 15 09:14:24.931987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:24.931937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:24.931987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:24.931937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:24.931987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:24.931946    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:24.932701 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:24.932180    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:24.932701 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:24.932573    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:24.933104 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:24.933074    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:25.936309 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:25.936134    2579 generic.go:356] "Generic (PLEG): container finished" podID="82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03" containerID="3e218cff62b4a07523a70e94f0460ef22539c3584dbeb2b7fb417da401a929ea" exitCode=0
Sep 15 09:14:25.936309 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:25.936216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psth7" event={"ID":"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03","Type":"ContainerDied","Data":"3e218cff62b4a07523a70e94f0460ef22539c3584dbeb2b7fb417da401a929ea"}
Sep 15 09:14:26.669688 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:26.669568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:26.669688 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:26.669641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:26.669688 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:26.669651    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:26.669917 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:26.669752    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:26.669917 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:26.669808    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:26.670015 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:26.669951    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:26.941528 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:26.941499    2579 generic.go:356] "Generic (PLEG): container finished" podID="82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03" containerID="34a2f851b81b3cdd870989c8055033e75b8723f37e61126ed704db44cd18c9ec" exitCode=0
Sep 15 09:14:26.941856 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:26.941543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psth7" event={"ID":"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03","Type":"ContainerDied","Data":"34a2f851b81b3cdd870989c8055033e75b8723f37e61126ed704db44cd18c9ec"}
Sep 15 09:14:28.669410 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:28.669380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:28.669806 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:28.669387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:28.669806 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:28.669495    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzmvm" podUID="26f8b6a1-afde-4839-9886-05fc1b9f1159"
Sep 15 09:14:28.669806 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:28.669585    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2pn5m" podUID="22d672a0-f79d-40fe-95e4-641910da6e65"
Sep 15 09:14:28.669806 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:28.669389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:28.669806 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:28.669689    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szh7l" podUID="92876d96-e3e2-4b10-ad5d-067e6ea18d22"
Sep 15 09:14:30.328617 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.328585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:30.328987 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.328737    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:30.328987 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.328809    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs podName:26f8b6a1-afde-4839-9886-05fc1b9f1159 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:02.328791298 +0000 UTC m=+66.158583197 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs") pod "network-metrics-daemon-kzmvm" (UID: "26f8b6a1-afde-4839-9886-05fc1b9f1159") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:14:30.429318 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.429263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:30.429486 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.429428    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:30.429548 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.429501    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret podName:92876d96-e3e2-4b10-ad5d-067e6ea18d22 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:02.42948192 +0000 UTC m=+66.259273818 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret") pod "global-pull-secret-syncer-szh7l" (UID: "92876d96-e3e2-4b10-ad5d-067e6ea18d22") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:14:30.506657 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.506623    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeReady"
Sep 15 09:14:30.506790 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.506779    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 09:14:30.530406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.530380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:30.530608 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.530588    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:14:30.530666 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.530614    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:14:30.530666 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.530626    2579 projected.go:196] Error preparing data for projected volume kube-api-access-x98sq for pod openshift-network-diagnostics/network-check-target-2pn5m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:30.530741 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.530677    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq podName:22d672a0-f79d-40fe-95e4-641910da6e65 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:02.530663388 +0000 UTC m=+66.360455269 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x98sq" (UniqueName: "kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq") pod "network-check-target-2pn5m" (UID: "22d672a0-f79d-40fe-95e4-641910da6e65") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:14:30.555620 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.555556    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"]
Sep 15 09:14:30.581350 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.581312    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vz4v2"]
Sep 15 09:14:30.581473 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.581435    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.585384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.585360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 09:14:30.585533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.585403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 09:14:30.585533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.585420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 09:14:30.585865 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.585843    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pc6pr\""
Sep 15 09:14:30.592649 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.592631    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 09:14:30.596384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.596363    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pt8lk"]
Sep 15 09:14:30.596482 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.596388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vz4v2"
Sep 15 09:14:30.598625 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.598609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 09:14:30.598800 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.598778    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 09:14:30.599044 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.599028    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wcvd7\""
Sep 15 09:14:30.599143 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.599125    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 09:14:30.609529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.609499    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vz4v2"]
Sep 15 09:14:30.609619 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.609534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"]
Sep 15 09:14:30.609619 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.609554    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pt8lk"]
Sep 15 09:14:30.609910 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.609885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.613179 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.613157    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 09:14:30.620461 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.620233    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 09:14:30.620461 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.620293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r95hw\""
Sep 15 09:14:30.669593 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.669574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:14:30.669593 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.669589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:14:30.669902 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.669881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:14:30.672123 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.671977    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 09:14:30.672123 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.672037    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 09:14:30.672123 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.672051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6rrz2\""
Sep 15 09:14:30.672324 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.672131    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ddxwx\""
Sep 15 09:14:30.672324 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.672291    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 09:14:30.672498 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.672469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 09:14:30.732688 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.732662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-certificates\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.732802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.732703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-bound-sa-token\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.732802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.732747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/766cc3d3-fef6-44d3-aeaa-74cf48211665-ca-trust-extracted\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.732802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.732769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3c8d2b7-0f7a-42fd-b51a-29062eed16a5-cert\") pod \"ingress-canary-vz4v2\" (UID: \"f3c8d2b7-0f7a-42fd-b51a-29062eed16a5\") " pod="openshift-ingress-canary/ingress-canary-vz4v2"
Sep 15 09:14:30.732802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.732796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.733011 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.732863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-installation-pull-secrets\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.733011 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.732890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-image-registry-private-configuration\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.733011 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.732955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d07861-7136-4a13-93a8-d959c6632229-config-volume\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.733011 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.732981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d07861-7136-4a13-93a8-d959c6632229-tmp-dir\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.733011 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.733002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmr8j\" (UniqueName: \"kubernetes.io/projected/a8d07861-7136-4a13-93a8-d959c6632229-kube-api-access-mmr8j\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.733234 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.733128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-trusted-ca\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.733234 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.733145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lnb\" (UniqueName: \"kubernetes.io/projected/f3c8d2b7-0f7a-42fd-b51a-29062eed16a5-kube-api-access-p5lnb\") pod \"ingress-canary-vz4v2\" (UID: \"f3c8d2b7-0f7a-42fd-b51a-29062eed16a5\") " pod="openshift-ingress-canary/ingress-canary-vz4v2"
Sep 15 09:14:30.733234 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.733168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.733234 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.733186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2h8s\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-kube-api-access-w2h8s\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.834043 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.833965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-image-registry-private-configuration\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.834043 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d07861-7136-4a13-93a8-d959c6632229-config-volume\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.834043 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d07861-7136-4a13-93a8-d959c6632229-tmp-dir\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.834306 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr8j\" (UniqueName: \"kubernetes.io/projected/a8d07861-7136-4a13-93a8-d959c6632229-kube-api-access-mmr8j\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.834306 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-trusted-ca\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.834306 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lnb\" (UniqueName: \"kubernetes.io/projected/f3c8d2b7-0f7a-42fd-b51a-29062eed16a5-kube-api-access-p5lnb\") pod \"ingress-canary-vz4v2\" (UID: \"f3c8d2b7-0f7a-42fd-b51a-29062eed16a5\") " pod="openshift-ingress-canary/ingress-canary-vz4v2"
Sep 15 09:14:30.834502 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.834556 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2h8s\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-kube-api-access-w2h8s\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.834556 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-certificates\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.834650 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-bound-sa-token\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.834650 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d07861-7136-4a13-93a8-d959c6632229-tmp-dir\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.834650 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/766cc3d3-fef6-44d3-aeaa-74cf48211665-ca-trust-extracted\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.834650 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3c8d2b7-0f7a-42fd-b51a-29062eed16a5-cert\") pod \"ingress-canary-vz4v2\" (UID: \"f3c8d2b7-0f7a-42fd-b51a-29062eed16a5\") " pod="openshift-ingress-canary/ingress-canary-vz4v2"
Sep 15 09:14:30.834834 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.834834 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-installation-pull-secrets\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.834950 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.834931    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:14:30.834998 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.834984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-certificates\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.835049 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.835012    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls podName:a8d07861-7136-4a13-93a8-d959c6632229 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:31.334991363 +0000 UTC m=+35.164783261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls") pod "dns-default-pt8lk" (UID: "a8d07861-7136-4a13-93a8-d959c6632229") : secret "dns-default-metrics-tls" not found
Sep 15 09:14:30.835127 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.835104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-trusted-ca\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.835207 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.835191    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:14:30.835265 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.835210    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb: secret "image-registry-tls" not found
Sep 15 09:14:30.835265 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.835244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/766cc3d3-fef6-44d3-aeaa-74cf48211665-ca-trust-extracted\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.835394 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:30.835265    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls podName:766cc3d3-fef6-44d3-aeaa-74cf48211665 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:31.335249461 +0000 UTC m=+35.165041350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls") pod "image-registry-5b7dc7bbc8-g4zpb" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665") : secret "image-registry-tls" not found
Sep 15 09:14:30.835586 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.835565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d07861-7136-4a13-93a8-d959c6632229-config-volume\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.838936 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.838914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3c8d2b7-0f7a-42fd-b51a-29062eed16a5-cert\") pod \"ingress-canary-vz4v2\" (UID: \"f3c8d2b7-0f7a-42fd-b51a-29062eed16a5\") " pod="openshift-ingress-canary/ingress-canary-vz4v2"
Sep 15 09:14:30.839040 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.838969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-installation-pull-secrets\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.839407 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.839387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-image-registry-private-configuration\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.846294 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.846249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-bound-sa-token\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.846424 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.846403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmr8j\" (UniqueName: \"kubernetes.io/projected/a8d07861-7136-4a13-93a8-d959c6632229-kube-api-access-mmr8j\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:30.846920 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.846900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2h8s\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-kube-api-access-w2h8s\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:30.846990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.846945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lnb\" (UniqueName: \"kubernetes.io/projected/f3c8d2b7-0f7a-42fd-b51a-29062eed16a5-kube-api-access-p5lnb\") pod \"ingress-canary-vz4v2\" (UID: \"f3c8d2b7-0f7a-42fd-b51a-29062eed16a5\") " pod="openshift-ingress-canary/ingress-canary-vz4v2"
Sep 15 09:14:30.906961 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:30.906936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vz4v2"
Sep 15 09:14:31.337214 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:31.337183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:31.337577 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:31.337265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:31.337577 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:31.337371    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:14:31.337577 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:31.337389    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb: secret "image-registry-tls" not found
Sep 15 09:14:31.337577 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:31.337371    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:14:31.337577 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:31.337448    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls podName:766cc3d3-fef6-44d3-aeaa-74cf48211665 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:32.337433584 +0000 UTC m=+36.167225469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls") pod "image-registry-5b7dc7bbc8-g4zpb" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665") : secret "image-registry-tls" not found
Sep 15 09:14:31.337577 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:31.337462    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls podName:a8d07861-7136-4a13-93a8-d959c6632229 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:32.337456514 +0000 UTC m=+36.167248395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls") pod "dns-default-pt8lk" (UID: "a8d07861-7136-4a13-93a8-d959c6632229") : secret "dns-default-metrics-tls" not found
Sep 15 09:14:31.645772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:31.645630    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vz4v2"]
Sep 15 09:14:31.648838 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:14:31.648817    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c8d2b7_0f7a_42fd_b51a_29062eed16a5.slice/crio-a59789c6644688aacc6fd5bbe17f8aba942cbc4cb0edb6c8c1bbb8aafe898a89 WatchSource:0}: Error finding container a59789c6644688aacc6fd5bbe17f8aba942cbc4cb0edb6c8c1bbb8aafe898a89: Status 404 returned error can't find the container with id a59789c6644688aacc6fd5bbe17f8aba942cbc4cb0edb6c8c1bbb8aafe898a89
Sep 15 09:14:31.955003 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:31.954961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vz4v2" event={"ID":"f3c8d2b7-0f7a-42fd-b51a-29062eed16a5","Type":"ContainerStarted","Data":"a59789c6644688aacc6fd5bbe17f8aba942cbc4cb0edb6c8c1bbb8aafe898a89"}
Sep 15 09:14:31.957588 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:31.957565    2579 generic.go:356] "Generic (PLEG): container finished" podID="82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03" containerID="7a00c70475e810746b3ebf75a35160aaff62f247e9939979d6063f88d8c4a9df" exitCode=0
Sep 15 09:14:31.957685 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:31.957605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psth7" event={"ID":"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03","Type":"ContainerDied","Data":"7a00c70475e810746b3ebf75a35160aaff62f247e9939979d6063f88d8c4a9df"}
Sep 15 09:14:32.343704 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:32.343627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:32.343704 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:32.343682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:32.344393 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:32.343783    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:14:32.344393 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:32.343785    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:14:32.344393 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:32.343859    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls podName:a8d07861-7136-4a13-93a8-d959c6632229 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:34.343838062 +0000 UTC m=+38.173629962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls") pod "dns-default-pt8lk" (UID: "a8d07861-7136-4a13-93a8-d959c6632229") : secret "dns-default-metrics-tls" not found
Sep 15 09:14:32.344393 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:32.343795    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb: secret "image-registry-tls" not found
Sep 15 09:14:32.344393 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:32.343934    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls podName:766cc3d3-fef6-44d3-aeaa-74cf48211665 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:34.343921925 +0000 UTC m=+38.173713822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls") pod "image-registry-5b7dc7bbc8-g4zpb" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665") : secret "image-registry-tls" not found
Sep 15 09:14:32.964397 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:32.964359    2579 generic.go:356] "Generic (PLEG): container finished" podID="82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03" containerID="07e4b0d2e4b52d81f8531115b9fbf7be01d1b259c475f76ee2ec2a87a1f8d01f" exitCode=0
Sep 15 09:14:32.964565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:32.964405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psth7" event={"ID":"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03","Type":"ContainerDied","Data":"07e4b0d2e4b52d81f8531115b9fbf7be01d1b259c475f76ee2ec2a87a1f8d01f"}
Sep 15 09:14:33.972216 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:33.972139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psth7" event={"ID":"82bcc1b7-3ce1-49b3-9e35-dcaa1f9c3f03","Type":"ContainerStarted","Data":"ac1c051ed698f2676ce7cab0cb4820bd191c29dc5016da4370504f301a191f3c"}
Sep 15 09:14:33.993963 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:33.993913    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-psth7" podStartSLOduration=8.204921858 podStartE2EDuration="36.993896788s" totalImagesPullingTime="28.78897493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:33.992687582 +0000 UTC m=+37.822479484" watchObservedRunningTime="2026-09-15 09:14:33.993896788 +0000 UTC m=+37.823688690"
Sep 15 09:14:34.359311 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:34.359285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:34.359444 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:34.359330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:34.359494 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:34.359436    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:14:34.359494 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:34.359449    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:14:34.359494 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:34.359459    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb: secret "image-registry-tls" not found
Sep 15 09:14:34.359593 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:34.359499    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls podName:a8d07861-7136-4a13-93a8-d959c6632229 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:38.359484703 +0000 UTC m=+42.189276606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls") pod "dns-default-pt8lk" (UID: "a8d07861-7136-4a13-93a8-d959c6632229") : secret "dns-default-metrics-tls" not found
Sep 15 09:14:34.359593 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:34.359513    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls podName:766cc3d3-fef6-44d3-aeaa-74cf48211665 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:38.359506993 +0000 UTC m=+42.189298873 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls") pod "image-registry-5b7dc7bbc8-g4zpb" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665") : secret "image-registry-tls" not found
Sep 15 09:14:34.976168 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:34.975970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vz4v2" event={"ID":"f3c8d2b7-0f7a-42fd-b51a-29062eed16a5","Type":"ContainerStarted","Data":"d27c7918fb02ab96eedeba2bd4f150b4d65393d64bebe966690a95c52b7db016"}
Sep 15 09:14:34.991038 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:34.990994    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vz4v2" podStartSLOduration=2.342815914 podStartE2EDuration="4.990980454s" totalImagesPullingTime="2.64816454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:34.990629613 +0000 UTC m=+38.820421517" watchObservedRunningTime="2026-09-15 09:14:34.990980454 +0000 UTC m=+38.820772356"
Sep 15 09:14:38.388818 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:38.388781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:38.389223 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:38.388850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:38.389223 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:38.388931    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:14:38.389223 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:38.388939    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:14:38.389223 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:38.388956    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb: secret "image-registry-tls" not found
Sep 15 09:14:38.389223 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:38.388981    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls podName:a8d07861-7136-4a13-93a8-d959c6632229 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:46.388968697 +0000 UTC m=+50.218760579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls") pod "dns-default-pt8lk" (UID: "a8d07861-7136-4a13-93a8-d959c6632229") : secret "dns-default-metrics-tls" not found
Sep 15 09:14:38.389223 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:38.389000    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls podName:766cc3d3-fef6-44d3-aeaa-74cf48211665 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:46.388986848 +0000 UTC m=+50.218778729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls") pod "image-registry-5b7dc7bbc8-g4zpb" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665") : secret "image-registry-tls" not found
Sep 15 09:14:40.147225 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.147191    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"]
Sep 15 09:14:40.174097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.174078    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"]
Sep 15 09:14:40.174206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.174159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:40.176271 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.176248    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ddcw6\""
Sep 15 09:14:40.176271 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.176268    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 09:14:40.176462 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.176248    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 09:14:40.302204 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.302182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:40.302314 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.302222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:40.403565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.403509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:40.403565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.403549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:40.403687 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:40.403658    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:14:40.403731 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:40.403721    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert podName:12ac97ae-6fdc-479b-9f49-1b3bec6717a9 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:40.903703727 +0000 UTC m=+44.733495628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-s5wgg" (UID: "12ac97ae-6fdc-479b-9f49-1b3bec6717a9") : secret "networking-console-plugin-cert" not found
Sep 15 09:14:40.404110 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.404094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:40.907639 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:40.907605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:40.907812 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:40.907721    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:14:40.907812 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:40.907780    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert podName:12ac97ae-6fdc-479b-9f49-1b3bec6717a9 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:41.907764367 +0000 UTC m=+45.737556249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-s5wgg" (UID: "12ac97ae-6fdc-479b-9f49-1b3bec6717a9") : secret "networking-console-plugin-cert" not found
Sep 15 09:14:41.914540 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:41.914504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:41.914976 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:41.914645    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:14:41.914976 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:41.914703    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert podName:12ac97ae-6fdc-479b-9f49-1b3bec6717a9 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:43.914687015 +0000 UTC m=+47.744478896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-s5wgg" (UID: "12ac97ae-6fdc-479b-9f49-1b3bec6717a9") : secret "networking-console-plugin-cert" not found
Sep 15 09:14:43.933509 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:43.933468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:43.933873 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:43.933627    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:14:43.933873 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:43.933705    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert podName:12ac97ae-6fdc-479b-9f49-1b3bec6717a9 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:47.93368945 +0000 UTC m=+51.763481331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-s5wgg" (UID: "12ac97ae-6fdc-479b-9f49-1b3bec6717a9") : secret "networking-console-plugin-cert" not found
Sep 15 09:14:46.448244 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:46.448211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:14:46.448626 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:46.448264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:14:46.448626 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:46.448390    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:14:46.448626 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:46.448394    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:14:46.448626 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:46.448483    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls podName:a8d07861-7136-4a13-93a8-d959c6632229 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:02.448463852 +0000 UTC m=+66.278255751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls") pod "dns-default-pt8lk" (UID: "a8d07861-7136-4a13-93a8-d959c6632229") : secret "dns-default-metrics-tls" not found
Sep 15 09:14:46.448626 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:46.448490    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb: secret "image-registry-tls" not found
Sep 15 09:14:46.448626 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:46.448534    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls podName:766cc3d3-fef6-44d3-aeaa-74cf48211665 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:02.448523284 +0000 UTC m=+66.278315165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls") pod "image-registry-5b7dc7bbc8-g4zpb" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665") : secret "image-registry-tls" not found
Sep 15 09:14:47.378250 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.378217    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5"]
Sep 15 09:14:47.387611 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.387591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5"
Sep 15 09:14:47.389681 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.389660    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 09:14:47.390351 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.390319    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 09:14:47.396681 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.396657    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5"]
Sep 15 09:14:47.398305 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.398288    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4jlpc\""
Sep 15 09:14:47.455411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.455384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslgk\" (UniqueName: \"kubernetes.io/projected/1bc192c2-9130-4b77-bb0f-5aa33c269203-kube-api-access-xslgk\") pod \"migrator-8b4d697c5-9nwb5\" (UID: \"1bc192c2-9130-4b77-bb0f-5aa33c269203\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5"
Sep 15 09:14:47.556514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.556484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslgk\" (UniqueName: \"kubernetes.io/projected/1bc192c2-9130-4b77-bb0f-5aa33c269203-kube-api-access-xslgk\") pod \"migrator-8b4d697c5-9nwb5\" (UID: \"1bc192c2-9130-4b77-bb0f-5aa33c269203\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5"
Sep 15 09:14:47.565310 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.565291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslgk\" (UniqueName: \"kubernetes.io/projected/1bc192c2-9130-4b77-bb0f-5aa33c269203-kube-api-access-xslgk\") pod \"migrator-8b4d697c5-9nwb5\" (UID: \"1bc192c2-9130-4b77-bb0f-5aa33c269203\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5"
Sep 15 09:14:47.691925 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.691902    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d7vlm"]
Sep 15 09:14:47.694730 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.694716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.695352 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.695312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5"
Sep 15 09:14:47.697048 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.697016    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 09:14:47.697403 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.697381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pkv2r\""
Sep 15 09:14:47.697811 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.697741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 09:14:47.697811 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.697768    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 09:14:47.697924 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.697839    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 09:14:47.706423 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.706405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7vlm"]
Sep 15 09:14:47.758657 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.758624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78mw\" (UniqueName: \"kubernetes.io/projected/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-api-access-m78mw\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.758778 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.758682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6260a04b-ea51-4bde-9d0a-797236eb14e5-crio-socket\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.758778 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.758737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.758893 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.758796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.758957 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.758908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6260a04b-ea51-4bde-9d0a-797236eb14e5-data-volume\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.859251 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.859220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6260a04b-ea51-4bde-9d0a-797236eb14e5-data-volume\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.859389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.859262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78mw\" (UniqueName: \"kubernetes.io/projected/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-api-access-m78mw\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.859389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.859293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6260a04b-ea51-4bde-9d0a-797236eb14e5-crio-socket\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.859389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.859326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.859389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.859377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.859629 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:47.859478    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 09:14:47.859629 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:47.859480    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:47.859629 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:47.859539    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:48.359523849 +0000 UTC m=+52.189315749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : configmap "kube-rbac-proxy" not found
Sep 15 09:14:47.859629 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.859544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6260a04b-ea51-4bde-9d0a-797236eb14e5-crio-socket\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.859629 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:47.859555    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:48.359547982 +0000 UTC m=+52.189339863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:47.859629 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.859535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6260a04b-ea51-4bde-9d0a-797236eb14e5-data-volume\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.866876 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.866846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78mw\" (UniqueName: \"kubernetes.io/projected/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-api-access-m78mw\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:47.959926 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.959871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:47.960021 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:47.959962    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:14:47.960021 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:47.960005    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert podName:12ac97ae-6fdc-479b-9f49-1b3bec6717a9 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:55.959991695 +0000 UTC m=+59.789783590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-s5wgg" (UID: "12ac97ae-6fdc-479b-9f49-1b3bec6717a9") : secret "networking-console-plugin-cert" not found
Sep 15 09:14:47.966328 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:47.966306    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5"]
Sep 15 09:14:47.969506 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:14:47.969477    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc192c2_9130_4b77_bb0f_5aa33c269203.slice/crio-59b2510ef1a7e9bfcc40b13a8d4144e943a11a9f8530437fcc54951cf2f4268f WatchSource:0}: Error finding container 59b2510ef1a7e9bfcc40b13a8d4144e943a11a9f8530437fcc54951cf2f4268f: Status 404 returned error can't find the container with id 59b2510ef1a7e9bfcc40b13a8d4144e943a11a9f8530437fcc54951cf2f4268f
Sep 15 09:14:48.006503 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:48.006474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5" event={"ID":"1bc192c2-9130-4b77-bb0f-5aa33c269203","Type":"ContainerStarted","Data":"59b2510ef1a7e9bfcc40b13a8d4144e943a11a9f8530437fcc54951cf2f4268f"}
Sep 15 09:14:48.362872 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:48.362791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:48.362872 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:48.362842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:48.363054 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:48.362927    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 09:14:48.363054 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:48.362939    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:48.363054 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:48.362977    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:49.362964688 +0000 UTC m=+53.192756568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : configmap "kube-rbac-proxy" not found
Sep 15 09:14:48.363054 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:48.362991    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:49.362983914 +0000 UTC m=+53.192775795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:49.371525 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:49.371490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:49.371923 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:49.371544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:49.371923 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:49.371642    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 09:14:49.371923 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:49.371665    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:49.371923 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:49.371692    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:51.371678895 +0000 UTC m=+55.201470776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : configmap "kube-rbac-proxy" not found
Sep 15 09:14:49.371923 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:49.371728    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:51.371710403 +0000 UTC m=+55.201502301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:49.874090 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:49.874066    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2vzlg_6b6c0984-533c-4e9e-a633-28abecd84a99/dns-node-resolver/0.log"
Sep 15 09:14:50.013560 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:50.013529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5" event={"ID":"1bc192c2-9130-4b77-bb0f-5aa33c269203","Type":"ContainerStarted","Data":"b20ce08057e11f86b7f1014d248e736131d06a3c03c0d03774edfacefe82e980"}
Sep 15 09:14:50.013701 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:50.013579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5" event={"ID":"1bc192c2-9130-4b77-bb0f-5aa33c269203","Type":"ContainerStarted","Data":"c744e395898f71dff9dc736e66ecd511e8a741d1e5e2be349a9fd6ab05336633"}
Sep 15 09:14:50.033008 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:50.032958    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9nwb5" podStartSLOduration=1.861110492 podStartE2EDuration="3.032940597s" totalImagesPullingTime="1.171830105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:50.032077266 +0000 UTC m=+53.861869168" watchObservedRunningTime="2026-09-15 09:14:50.032940597 +0000 UTC m=+53.862732538"
Sep 15 09:14:51.274317 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:51.274290    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vjcp6_f65720d5-a59e-4e28-a582-c4ae0b861511/node-ca/0.log"
Sep 15 09:14:51.386460 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:51.386435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:51.386592 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:51.386481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:51.386629 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:51.386610    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:51.386672 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:51.386662    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:55.386649762 +0000 UTC m=+59.216441662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:51.386721 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:51.386675    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 09:14:51.386767 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:51.386722    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:14:55.386710039 +0000 UTC m=+59.216501920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : configmap "kube-rbac-proxy" not found
Sep 15 09:14:51.875009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:51.874980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vz4v2_f3c8d2b7-0f7a-42fd-b51a-29062eed16a5/serve-healthcheck-canary/0.log"
Sep 15 09:14:52.075801 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:52.075770    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-9nwb5_1bc192c2-9130-4b77-bb0f-5aa33c269203/migrator/0.log"
Sep 15 09:14:52.275123 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:52.275090    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-9nwb5_1bc192c2-9130-4b77-bb0f-5aa33c269203/graceful-termination/0.log"
Sep 15 09:14:54.942741 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:54.942711    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7v2rw"
Sep 15 09:14:55.413220 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:55.413189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:55.413444 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:55.413246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:14:55.413444 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:55.413366    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 09:14:55.413444 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:55.413401    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:55.413444 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:55.413430    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:03.413416033 +0000 UTC m=+67.243207913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : configmap "kube-rbac-proxy" not found
Sep 15 09:14:55.413638 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:55.413478    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:03.41346033 +0000 UTC m=+67.243252228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:14:56.017637 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:14:56.017593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:14:56.018005 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:56.017717    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:14:56.018005 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:14:56.017782    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert podName:12ac97ae-6fdc-479b-9f49-1b3bec6717a9 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:12.017768606 +0000 UTC m=+75.847560487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-s5wgg" (UID: "12ac97ae-6fdc-479b-9f49-1b3bec6717a9") : secret "networking-console-plugin-cert" not found
Sep 15 09:15:02.366501 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.366464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:15:02.368660 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.368641    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 09:15:02.377562 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:15:02.377544    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 09:15:02.377620 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:15:02.377599    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs podName:26f8b6a1-afde-4839-9886-05fc1b9f1159 nodeName:}" failed. No retries permitted until 2026-09-15 09:16:06.377582046 +0000 UTC m=+130.207373949 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs") pod "network-metrics-daemon-kzmvm" (UID: "26f8b6a1-afde-4839-9886-05fc1b9f1159") : secret "metrics-daemon-secret" not found
Sep 15 09:15:02.467471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.467446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:15:02.467559 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.467486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:15:02.467559 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.467505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:15:02.469745 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.469725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"image-registry-5b7dc7bbc8-g4zpb\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") " pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:15:02.469835 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.469758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8d07861-7136-4a13-93a8-d959c6632229-metrics-tls\") pod \"dns-default-pt8lk\" (UID: \"a8d07861-7136-4a13-93a8-d959c6632229\") " pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:15:02.469874 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.469842    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 09:15:02.480207 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.480180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92876d96-e3e2-4b10-ad5d-067e6ea18d22-original-pull-secret\") pod \"global-pull-secret-syncer-szh7l\" (UID: \"92876d96-e3e2-4b10-ad5d-067e6ea18d22\") " pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:15:02.567897 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.567871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:15:02.570067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.570052    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 09:15:02.580936 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.580921    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 09:15:02.591500 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.591479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x98sq\" (UniqueName: \"kubernetes.io/projected/22d672a0-f79d-40fe-95e4-641910da6e65-kube-api-access-x98sq\") pod \"network-check-target-2pn5m\" (UID: \"22d672a0-f79d-40fe-95e4-641910da6e65\") " pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:15:02.694973 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.694950    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pc6pr\""
Sep 15 09:15:02.702648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.702632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:15:02.729183 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.729164    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r95hw\""
Sep 15 09:15:02.737981 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.737952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:15:02.780141 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.780116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szh7l"
Sep 15 09:15:02.789709 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.789485    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6rrz2\""
Sep 15 09:15:02.797652 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.797621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:15:02.975745 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:02.975724    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"]
Sep 15 09:15:02.978039 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:02.978013    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766cc3d3_fef6_44d3_aeaa_74cf48211665.slice/crio-dfae11e7a8ffbd59a10eb52866695914192b10c98da3397ae29e021028a86e73 WatchSource:0}: Error finding container dfae11e7a8ffbd59a10eb52866695914192b10c98da3397ae29e021028a86e73: Status 404 returned error can't find the container with id dfae11e7a8ffbd59a10eb52866695914192b10c98da3397ae29e021028a86e73
Sep 15 09:15:03.015773 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.015293    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pt8lk"]
Sep 15 09:15:03.018110 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:03.018089    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d07861_7136_4a13_93a8_d959c6632229.slice/crio-ad69c465f471e109835c3a107e8d3aeb5a29eef4a672f702c8de6e8356b211c6 WatchSource:0}: Error finding container ad69c465f471e109835c3a107e8d3aeb5a29eef4a672f702c8de6e8356b211c6: Status 404 returned error can't find the container with id ad69c465f471e109835c3a107e8d3aeb5a29eef4a672f702c8de6e8356b211c6
Sep 15 09:15:03.047140 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.047113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pt8lk" event={"ID":"a8d07861-7136-4a13-93a8-d959c6632229","Type":"ContainerStarted","Data":"ad69c465f471e109835c3a107e8d3aeb5a29eef4a672f702c8de6e8356b211c6"}
Sep 15 09:15:03.048513 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.048487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb" event={"ID":"766cc3d3-fef6-44d3-aeaa-74cf48211665","Type":"ContainerStarted","Data":"0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6"}
Sep 15 09:15:03.048592 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.048528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb" event={"ID":"766cc3d3-fef6-44d3-aeaa-74cf48211665","Type":"ContainerStarted","Data":"dfae11e7a8ffbd59a10eb52866695914192b10c98da3397ae29e021028a86e73"}
Sep 15 09:15:03.048638 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.048621    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:15:03.051877 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.051856    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-szh7l"]
Sep 15 09:15:03.054838 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:03.054816    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92876d96_e3e2_4b10_ad5d_067e6ea18d22.slice/crio-3d42ed80458eb0004457c4682b495c6fb820f203ad1ff9460f8a5dae37663677 WatchSource:0}: Error finding container 3d42ed80458eb0004457c4682b495c6fb820f203ad1ff9460f8a5dae37663677: Status 404 returned error can't find the container with id 3d42ed80458eb0004457c4682b495c6fb820f203ad1ff9460f8a5dae37663677
Sep 15 09:15:03.068567 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.068508    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb" podStartSLOduration=35.06848932 podStartE2EDuration="35.06848932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:03.06777467 +0000 UTC m=+66.897566572" watchObservedRunningTime="2026-09-15 09:15:03.06848932 +0000 UTC m=+66.898281220"
Sep 15 09:15:03.071758 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.071724    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2pn5m"]
Sep 15 09:15:03.074810 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:03.074783    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d672a0_f79d_40fe_95e4_641910da6e65.slice/crio-293fa633a7ca54632bd823a4e9f8cd3da76509f401c2d717d4ac7b2384db6bff WatchSource:0}: Error finding container 293fa633a7ca54632bd823a4e9f8cd3da76509f401c2d717d4ac7b2384db6bff: Status 404 returned error can't find the container with id 293fa633a7ca54632bd823a4e9f8cd3da76509f401c2d717d4ac7b2384db6bff
Sep 15 09:15:03.476764 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.476735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:15:03.477189 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.476782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:15:03.477189 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:15:03.476918    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:15:03.477189 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:15:03.477015    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls podName:6260a04b-ea51-4bde-9d0a-797236eb14e5 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:19.476991728 +0000 UTC m=+83.306783631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7vlm" (UID: "6260a04b-ea51-4bde-9d0a-797236eb14e5") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:15:03.477425 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:03.477220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6260a04b-ea51-4bde-9d0a-797236eb14e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:15:04.053156 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:04.053085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-szh7l" event={"ID":"92876d96-e3e2-4b10-ad5d-067e6ea18d22","Type":"ContainerStarted","Data":"3d42ed80458eb0004457c4682b495c6fb820f203ad1ff9460f8a5dae37663677"}
Sep 15 09:15:04.055620 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:04.055561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2pn5m" event={"ID":"22d672a0-f79d-40fe-95e4-641910da6e65","Type":"ContainerStarted","Data":"293fa633a7ca54632bd823a4e9f8cd3da76509f401c2d717d4ac7b2384db6bff"}
Sep 15 09:15:06.066134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:06.066087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pt8lk" event={"ID":"a8d07861-7136-4a13-93a8-d959c6632229","Type":"ContainerStarted","Data":"788c1de40dcd384ef2ee4f83fd95f82d730f51c50f9d884773daa539419715ab"}
Sep 15 09:15:06.066614 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:06.066169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pt8lk" event={"ID":"a8d07861-7136-4a13-93a8-d959c6632229","Type":"ContainerStarted","Data":"3a3ca9f86fe3f1539b9acca9e7378bd1119d2e2674e62336237cdebef522c6e5"}
Sep 15 09:15:06.066614 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:06.066218    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:15:06.082306 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:06.082268    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pt8lk" podStartSLOduration=33.647207603 podStartE2EDuration="36.082255456s" totalImagesPullingTime="2.435047853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:06.080616626 +0000 UTC m=+69.910408528" watchObservedRunningTime="2026-09-15 09:15:06.082255456 +0000 UTC m=+69.912047360"
Sep 15 09:15:09.078761 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:09.078721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-szh7l" event={"ID":"92876d96-e3e2-4b10-ad5d-067e6ea18d22","Type":"ContainerStarted","Data":"a0951ea5f42b4cf4c65ff35435cdbc845382750626c1bb0de4df3fa76f4a80f8"}
Sep 15 09:15:09.080290 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:09.080258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2pn5m" event={"ID":"22d672a0-f79d-40fe-95e4-641910da6e65","Type":"ContainerStarted","Data":"04b8df6851e2d702dda3bf240add96488197fb2e4764527d488979497cc40eae"}
Sep 15 09:15:09.080415 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:09.080392    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:15:09.092297 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:09.092244    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-szh7l" podStartSLOduration=66.86222835 podStartE2EDuration="1m12.092225915s" totalImagesPullingTime="5.229997565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:09.091944576 +0000 UTC m=+72.921736479" watchObservedRunningTime="2026-09-15 09:15:09.092225915 +0000 UTC m=+72.922017820"
Sep 15 09:15:09.105003 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:09.104961    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2pn5m" podStartSLOduration=66.900258779 podStartE2EDuration="1m12.104947421s" totalImagesPullingTime="5.204688642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:09.104815823 +0000 UTC m=+72.934607725" watchObservedRunningTime="2026-09-15 09:15:09.104947421 +0000 UTC m=+72.934739323"
Sep 15 09:15:12.043476 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:12.043430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:15:12.045885 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:12.045863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12ac97ae-6fdc-479b-9f49-1b3bec6717a9-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-s5wgg\" (UID: \"12ac97ae-6fdc-479b-9f49-1b3bec6717a9\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:15:12.285278 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:12.285250    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ddcw6\""
Sep 15 09:15:12.293043 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:12.293026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"
Sep 15 09:15:12.559219 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:12.559147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg"]
Sep 15 09:15:12.562738 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:12.562708    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ac97ae_6fdc_479b_9f49_1b3bec6717a9.slice/crio-1c08eb5898f6f497207aa641c156dfbf635ee3c111cd8089718651adcfb13df6 WatchSource:0}: Error finding container 1c08eb5898f6f497207aa641c156dfbf635ee3c111cd8089718651adcfb13df6: Status 404 returned error can't find the container with id 1c08eb5898f6f497207aa641c156dfbf635ee3c111cd8089718651adcfb13df6
Sep 15 09:15:13.094665 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:13.094628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg" event={"ID":"12ac97ae-6fdc-479b-9f49-1b3bec6717a9","Type":"ContainerStarted","Data":"1c08eb5898f6f497207aa641c156dfbf635ee3c111cd8089718651adcfb13df6"}
Sep 15 09:15:14.099822 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:14.099747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg" event={"ID":"12ac97ae-6fdc-479b-9f49-1b3bec6717a9","Type":"ContainerStarted","Data":"9bb2d23475a0c1885037760cdba32d2e610776003c7defdbac140b9da203eb89"}
Sep 15 09:15:14.115896 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:14.115854    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-s5wgg" podStartSLOduration=32.954031879 podStartE2EDuration="34.115837601s" totalImagesPullingTime="1.161805722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:14.11504187 +0000 UTC m=+77.944833773" watchObservedRunningTime="2026-09-15 09:15:14.115837601 +0000 UTC m=+77.945629503"
Sep 15 09:15:15.305166 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:15.305131    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"]
Sep 15 09:15:16.072348 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:16.072314    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pt8lk"
Sep 15 09:15:19.496301 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:19.496166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:15:19.498418 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:19.498397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6260a04b-ea51-4bde-9d0a-797236eb14e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7vlm\" (UID: \"6260a04b-ea51-4bde-9d0a-797236eb14e5\") " pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:15:19.520042 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:19.520022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pkv2r\""
Sep 15 09:15:19.527990 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:19.527974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7vlm"
Sep 15 09:15:19.795267 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:19.795190    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7vlm"]
Sep 15 09:15:19.798062 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:19.798036    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6260a04b_ea51_4bde_9d0a_797236eb14e5.slice/crio-bc6c728545fee1781e8490ab6845b3f8000fcf783dd22b7a7ad087462d637f1b WatchSource:0}: Error finding container bc6c728545fee1781e8490ab6845b3f8000fcf783dd22b7a7ad087462d637f1b: Status 404 returned error can't find the container with id bc6c728545fee1781e8490ab6845b3f8000fcf783dd22b7a7ad087462d637f1b
Sep 15 09:15:20.118363 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:20.118265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7vlm" event={"ID":"6260a04b-ea51-4bde-9d0a-797236eb14e5","Type":"ContainerStarted","Data":"bb6175045511915762edefe83a6e742177657a36fd8fa63eb7a8e3018ad5e975"}
Sep 15 09:15:20.118363 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:20.118321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7vlm" event={"ID":"6260a04b-ea51-4bde-9d0a-797236eb14e5","Type":"ContainerStarted","Data":"bc6c728545fee1781e8490ab6845b3f8000fcf783dd22b7a7ad087462d637f1b"}
Sep 15 09:15:21.123194 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:21.123119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7vlm" event={"ID":"6260a04b-ea51-4bde-9d0a-797236eb14e5","Type":"ContainerStarted","Data":"ff5cb0e46bb825ca8e1cb239adfdf9c9c118b07e06a8f62739302bb1da2c9147"}
Sep 15 09:15:23.131685 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:23.131651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7vlm" event={"ID":"6260a04b-ea51-4bde-9d0a-797236eb14e5","Type":"ContainerStarted","Data":"adb124ef665e7d3fc3118b678d7b7952c5ec9fd0225df9a14257a1bd0ba91321"}
Sep 15 09:15:23.153760 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:23.153717    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d7vlm" podStartSLOduration=33.670765897 podStartE2EDuration="36.153702434s" totalImagesPullingTime="2.482936537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:14:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:23.153075689 +0000 UTC m=+86.982867604" watchObservedRunningTime="2026-09-15 09:15:23.153702434 +0000 UTC m=+86.983494337"
Sep 15 09:15:25.311514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:25.311487    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:15:37.643669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:37.643636    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"]
Sep 15 09:15:37.647850 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:37.647833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"
Sep 15 09:15:37.651035 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:37.651018    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 15 09:15:37.651091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:37.651039    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mbbk6\""
Sep 15 09:15:37.656250 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:37.656230    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"]
Sep 15 09:15:37.721979 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:37.721955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f64a93cc-ed79-48bf-bb1f-7b1bed808b83-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-sdv9z\" (UID: \"f64a93cc-ed79-48bf-bb1f-7b1bed808b83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"
Sep 15 09:15:37.822302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:37.822276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f64a93cc-ed79-48bf-bb1f-7b1bed808b83-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-sdv9z\" (UID: \"f64a93cc-ed79-48bf-bb1f-7b1bed808b83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"
Sep 15 09:15:37.822416 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:15:37.822385    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 15 09:15:37.822463 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:15:37.822444    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f64a93cc-ed79-48bf-bb1f-7b1bed808b83-tls-certificates podName:f64a93cc-ed79-48bf-bb1f-7b1bed808b83 nodeName:}" failed. No retries permitted until 2026-09-15 09:15:38.322428721 +0000 UTC m=+102.152220614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f64a93cc-ed79-48bf-bb1f-7b1bed808b83-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-sdv9z" (UID: "f64a93cc-ed79-48bf-bb1f-7b1bed808b83") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 15 09:15:38.325026 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:38.324991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f64a93cc-ed79-48bf-bb1f-7b1bed808b83-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-sdv9z\" (UID: \"f64a93cc-ed79-48bf-bb1f-7b1bed808b83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"
Sep 15 09:15:38.327360 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:38.327320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f64a93cc-ed79-48bf-bb1f-7b1bed808b83-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-sdv9z\" (UID: \"f64a93cc-ed79-48bf-bb1f-7b1bed808b83\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"
Sep 15 09:15:38.557071 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:38.557039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"
Sep 15 09:15:38.822059 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:38.822025    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"]
Sep 15 09:15:38.825384 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:38.825362    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64a93cc_ed79_48bf_bb1f_7b1bed808b83.slice/crio-e94f06d24c10f04b26dd98d6f890f4a94c5bf139332274aa98fc0a1b1659c855 WatchSource:0}: Error finding container e94f06d24c10f04b26dd98d6f890f4a94c5bf139332274aa98fc0a1b1659c855: Status 404 returned error can't find the container with id e94f06d24c10f04b26dd98d6f890f4a94c5bf139332274aa98fc0a1b1659c855
Sep 15 09:15:39.180291 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:39.180253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z" event={"ID":"f64a93cc-ed79-48bf-bb1f-7b1bed808b83","Type":"ContainerStarted","Data":"e94f06d24c10f04b26dd98d6f890f4a94c5bf139332274aa98fc0a1b1659c855"}
Sep 15 09:15:40.087022 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.086993    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2pn5m"
Sep 15 09:15:40.323350 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.323295    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb" podUID="766cc3d3-fef6-44d3-aeaa-74cf48211665" containerName="registry" containerID="cri-o://0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6" gracePeriod=30
Sep 15 09:15:40.828257 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.828232    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:15:40.941299 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.941261    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-image-registry-private-configuration\") pod \"766cc3d3-fef6-44d3-aeaa-74cf48211665\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") "
Sep 15 09:15:40.941478 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.941314    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-bound-sa-token\") pod \"766cc3d3-fef6-44d3-aeaa-74cf48211665\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") "
Sep 15 09:15:40.941478 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.941411    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2h8s\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-kube-api-access-w2h8s\") pod \"766cc3d3-fef6-44d3-aeaa-74cf48211665\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") "
Sep 15 09:15:40.941478 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.941442    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-certificates\") pod \"766cc3d3-fef6-44d3-aeaa-74cf48211665\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") "
Sep 15 09:15:40.941648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.941490    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/766cc3d3-fef6-44d3-aeaa-74cf48211665-ca-trust-extracted\") pod \"766cc3d3-fef6-44d3-aeaa-74cf48211665\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") "
Sep 15 09:15:40.941648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.941523    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") pod \"766cc3d3-fef6-44d3-aeaa-74cf48211665\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") "
Sep 15 09:15:40.941648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.941556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-installation-pull-secrets\") pod \"766cc3d3-fef6-44d3-aeaa-74cf48211665\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") "
Sep 15 09:15:40.941648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.941585    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-trusted-ca\") pod \"766cc3d3-fef6-44d3-aeaa-74cf48211665\" (UID: \"766cc3d3-fef6-44d3-aeaa-74cf48211665\") "
Sep 15 09:15:40.941881 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.941838    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "766cc3d3-fef6-44d3-aeaa-74cf48211665" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:15:40.942082 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.942055    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "766cc3d3-fef6-44d3-aeaa-74cf48211665" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:15:40.943968 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.943924    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-kube-api-access-w2h8s" (OuterVolumeSpecName: "kube-api-access-w2h8s") pod "766cc3d3-fef6-44d3-aeaa-74cf48211665" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665"). InnerVolumeSpecName "kube-api-access-w2h8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:15:40.944073 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.944001    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "766cc3d3-fef6-44d3-aeaa-74cf48211665" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:15:40.944137 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.944103    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "766cc3d3-fef6-44d3-aeaa-74cf48211665" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:15:40.944278 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.944254    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "766cc3d3-fef6-44d3-aeaa-74cf48211665" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:15:40.944321 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.944252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "766cc3d3-fef6-44d3-aeaa-74cf48211665" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:15:40.950553 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:40.950529    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766cc3d3-fef6-44d3-aeaa-74cf48211665-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "766cc3d3-fef6-44d3-aeaa-74cf48211665" (UID: "766cc3d3-fef6-44d3-aeaa-74cf48211665"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:15:41.042891 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.042814    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-image-registry-private-configuration\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:15:41.042891 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.042844    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-bound-sa-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:15:41.042891 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.042856    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2h8s\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-kube-api-access-w2h8s\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:15:41.042891 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.042865    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-certificates\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:15:41.042891 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.042877    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/766cc3d3-fef6-44d3-aeaa-74cf48211665-ca-trust-extracted\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:15:41.042891 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.042889    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/766cc3d3-fef6-44d3-aeaa-74cf48211665-registry-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:15:41.042891 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.042898    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/766cc3d3-fef6-44d3-aeaa-74cf48211665-installation-pull-secrets\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:15:41.043196 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.042906    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/766cc3d3-fef6-44d3-aeaa-74cf48211665-trusted-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:15:41.188715 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.188678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z" event={"ID":"f64a93cc-ed79-48bf-bb1f-7b1bed808b83","Type":"ContainerStarted","Data":"494690c92a10e2f19c7e825e6cb7a7255a75c5ef372e99df99c7d6f007eee25f"}
Sep 15 09:15:41.189107 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.188858    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"
Sep 15 09:15:41.190032 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.190007    2579 generic.go:356] "Generic (PLEG): container finished" podID="766cc3d3-fef6-44d3-aeaa-74cf48211665" containerID="0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6" exitCode=0
Sep 15 09:15:41.190143 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.190059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb" event={"ID":"766cc3d3-fef6-44d3-aeaa-74cf48211665","Type":"ContainerDied","Data":"0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6"}
Sep 15 09:15:41.190143 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.190071    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"
Sep 15 09:15:41.190143 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.190090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb" event={"ID":"766cc3d3-fef6-44d3-aeaa-74cf48211665","Type":"ContainerDied","Data":"dfae11e7a8ffbd59a10eb52866695914192b10c98da3397ae29e021028a86e73"}
Sep 15 09:15:41.190143 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.190110    2579 scope.go:117] "RemoveContainer" containerID="0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6"
Sep 15 09:15:41.194223 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.194204    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z"
Sep 15 09:15:41.198436 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.198419    2579 scope.go:117] "RemoveContainer" containerID="0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6"
Sep 15 09:15:41.198690 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:15:41.198670    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6\": container with ID starting with 0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6 not found: ID does not exist" containerID="0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6"
Sep 15 09:15:41.198739 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.198697    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6"} err="failed to get container status \"0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6\": rpc error: code = NotFound desc = could not find container \"0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6\": container with ID starting with 0d8d0adbf9afee0313f2a3fb9c54ed38c3d9cac202fe7633690310e8d526f9f6 not found: ID does not exist"
Sep 15 09:15:41.202783 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.202747    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-sdv9z" podStartSLOduration=1.962421398 podStartE2EDuration="4.20273472s" totalImagesPullingTime="2.240313322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:41.201979701 +0000 UTC m=+105.031771604" watchObservedRunningTime="2026-09-15 09:15:41.20273472 +0000 UTC m=+105.032526656"
Sep 15 09:15:41.216958 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.216939    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"]
Sep 15 09:15:41.220377 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.220357    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b7dc7bbc8-g4zpb"]
Sep 15 09:15:41.695616 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.695583    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"]
Sep 15 09:15:41.695877 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.695860    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766cc3d3-fef6-44d3-aeaa-74cf48211665" containerName="registry"
Sep 15 09:15:41.695950 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.695880    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="766cc3d3-fef6-44d3-aeaa-74cf48211665" containerName="registry"
Sep 15 09:15:41.696016 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.695952    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="766cc3d3-fef6-44d3-aeaa-74cf48211665" containerName="registry"
Sep 15 09:15:41.725993 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.725963    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"]
Sep 15 09:15:41.726119 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.726079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.728766 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.728743    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dgb6z\""
Sep 15 09:15:41.729496 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.729472    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 09:15:41.729628 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.729517    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 09:15:41.729628 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.729528    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 09:15:41.729628 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.729616    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 09:15:41.729773 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.729759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 09:15:41.849394 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.849365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acb00edc-c2af-4c5f-9da5-86d74e899cc5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.849537 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.849402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tp2g\" (UniqueName: \"kubernetes.io/projected/acb00edc-c2af-4c5f-9da5-86d74e899cc5-kube-api-access-4tp2g\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.849537 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.849465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acb00edc-c2af-4c5f-9da5-86d74e899cc5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.849537 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.849504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acb00edc-c2af-4c5f-9da5-86d74e899cc5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.949827 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.949759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acb00edc-c2af-4c5f-9da5-86d74e899cc5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.949827 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.949792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tp2g\" (UniqueName: \"kubernetes.io/projected/acb00edc-c2af-4c5f-9da5-86d74e899cc5-kube-api-access-4tp2g\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.949827 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.949819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acb00edc-c2af-4c5f-9da5-86d74e899cc5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.950011 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.949872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acb00edc-c2af-4c5f-9da5-86d74e899cc5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.950564 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.950545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acb00edc-c2af-4c5f-9da5-86d74e899cc5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.952246 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.952223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/acb00edc-c2af-4c5f-9da5-86d74e899cc5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.952365 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.952256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acb00edc-c2af-4c5f-9da5-86d74e899cc5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:41.964059 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:41.964036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tp2g\" (UniqueName: \"kubernetes.io/projected/acb00edc-c2af-4c5f-9da5-86d74e899cc5-kube-api-access-4tp2g\") pod \"prometheus-operator-68bcd7f987-hbdtf\" (UID: \"acb00edc-c2af-4c5f-9da5-86d74e899cc5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:42.034829 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:42.034802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"
Sep 15 09:15:42.321132 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:42.321065    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf"]
Sep 15 09:15:42.323827 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:42.323802    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb00edc_c2af_4c5f_9da5_86d74e899cc5.slice/crio-07981c38ec65454a0beca85b7102d98ac3ea0c26c6467ededc9e8b009b7ffebf WatchSource:0}: Error finding container 07981c38ec65454a0beca85b7102d98ac3ea0c26c6467ededc9e8b009b7ffebf: Status 404 returned error can't find the container with id 07981c38ec65454a0beca85b7102d98ac3ea0c26c6467ededc9e8b009b7ffebf
Sep 15 09:15:42.672587 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:42.672555    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="766cc3d3-fef6-44d3-aeaa-74cf48211665" path="/var/lib/kubelet/pods/766cc3d3-fef6-44d3-aeaa-74cf48211665/volumes"
Sep 15 09:15:43.199276 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:43.199235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf" event={"ID":"acb00edc-c2af-4c5f-9da5-86d74e899cc5","Type":"ContainerStarted","Data":"07981c38ec65454a0beca85b7102d98ac3ea0c26c6467ededc9e8b009b7ffebf"}
Sep 15 09:15:45.207783 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:45.207740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf" event={"ID":"acb00edc-c2af-4c5f-9da5-86d74e899cc5","Type":"ContainerStarted","Data":"bc6193c26498187652011919c1747739bc8ff6d082db1b21020761ad5d10508f"}
Sep 15 09:15:45.208211 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:45.207799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf" event={"ID":"acb00edc-c2af-4c5f-9da5-86d74e899cc5","Type":"ContainerStarted","Data":"66b0077aec108bad9deb69612b52eb6d1501e3c4116f153f3e2426a99c94f6ea"}
Sep 15 09:15:45.224195 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:45.224149    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hbdtf" podStartSLOduration=2.244157511 podStartE2EDuration="4.224134617s" totalImagesPullingTime="1.979977106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:45.223571133 +0000 UTC m=+109.053363038" watchObservedRunningTime="2026-09-15 09:15:45.224134617 +0000 UTC m=+109.053926536"
Sep 15 09:15:47.057545 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.057511    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"]
Sep 15 09:15:47.061039 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.061022    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"]
Sep 15 09:15:47.061167 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.061149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.063775 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.063752    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 09:15:47.063926 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.063828    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 09:15:47.064036 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.064022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-snlgv\""
Sep 15 09:15:47.064036 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.064030    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.064529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.064507    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 09:15:47.066173 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.066154    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zlz97\""
Sep 15 09:15:47.066286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.066191    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 09:15:47.066528 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.066511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 09:15:47.075862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.075842    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"]
Sep 15 09:15:47.076573 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.076558    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"]
Sep 15 09:15:47.084518 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.084498    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-86n9r"]
Sep 15 09:15:47.087414 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.087396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.089506 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.089487    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 09:15:47.089598 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.089561    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 09:15:47.089658 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.089629    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dcwhg\""
Sep 15 09:15:47.089715 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.089692    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 09:15:47.186084 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fa2f9ec-477d-4607-a9ae-492efc929777-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.186206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-root\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.186206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e55de041-5230-4a5e-a871-ef60649a3c40-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.186206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-tls\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.186206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnwbz\" (UniqueName: \"kubernetes.io/projected/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-kube-api-access-xnwbz\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.186406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6js\" (UniqueName: \"kubernetes.io/projected/e55de041-5230-4a5e-a871-ef60649a3c40-kube-api-access-7l6js\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.186406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e55de041-5230-4a5e-a871-ef60649a3c40-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.186406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-sys\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.186406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e55de041-5230-4a5e-a871-ef60649a3c40-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.186406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-textfile\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.186406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-metrics-client-ca\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.186582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e55de041-5230-4a5e-a871-ef60649a3c40-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.186582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8f9\" (UniqueName: \"kubernetes.io/projected/1fa2f9ec-477d-4607-a9ae-492efc929777-kube-api-access-vz8f9\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.186582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fa2f9ec-477d-4607-a9ae-492efc929777-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.186582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e55de041-5230-4a5e-a871-ef60649a3c40-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.186582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fa2f9ec-477d-4607-a9ae-492efc929777-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.186582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.186582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-wtmp\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.186789 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.186599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-accelerators-collector-config\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.287647 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-metrics-client-ca\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.287754 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e55de041-5230-4a5e-a871-ef60649a3c40-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.287754 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8f9\" (UniqueName: \"kubernetes.io/projected/1fa2f9ec-477d-4607-a9ae-492efc929777-kube-api-access-vz8f9\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.287754 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fa2f9ec-477d-4607-a9ae-492efc929777-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.287754 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e55de041-5230-4a5e-a871-ef60649a3c40-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.287754 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fa2f9ec-477d-4607-a9ae-492efc929777-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.288024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-wtmp\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-accelerators-collector-config\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fa2f9ec-477d-4607-a9ae-492efc929777-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.288024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-root\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e55de041-5230-4a5e-a871-ef60649a3c40-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.288024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-tls\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnwbz\" (UniqueName: \"kubernetes.io/projected/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-kube-api-access-xnwbz\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.287997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6js\" (UniqueName: \"kubernetes.io/projected/e55de041-5230-4a5e-a871-ef60649a3c40-kube-api-access-7l6js\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.288504 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e55de041-5230-4a5e-a871-ef60649a3c40-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.288504 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-sys\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288504 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e55de041-5230-4a5e-a871-ef60649a3c40-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.288504 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-textfile\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288504 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-metrics-client-ca\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288504 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-textfile\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288792 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e55de041-5230-4a5e-a871-ef60649a3c40-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.288792 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fa2f9ec-477d-4607-a9ae-492efc929777-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.288792 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e55de041-5230-4a5e-a871-ef60649a3c40-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.288792 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-wtmp\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.288792 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-sys\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.289038 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-accelerators-collector-config\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.289038 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.288989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e55de041-5230-4a5e-a871-ef60649a3c40-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.289140 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.289060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-root\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.290890 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.290866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fa2f9ec-477d-4607-a9ae-492efc929777-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.291544 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.291519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e55de041-5230-4a5e-a871-ef60649a3c40-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.291544 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.291535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.291678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.291531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e55de041-5230-4a5e-a871-ef60649a3c40-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.291678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.291600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fa2f9ec-477d-4607-a9ae-492efc929777-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.291678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.291605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-node-exporter-tls\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.296610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.296585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6js\" (UniqueName: \"kubernetes.io/projected/e55de041-5230-4a5e-a871-ef60649a3c40-kube-api-access-7l6js\") pod \"kube-state-metrics-59fbdd77d-w2hdj\" (UID: \"e55de041-5230-4a5e-a871-ef60649a3c40\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.296937 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.296913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnwbz\" (UniqueName: \"kubernetes.io/projected/d72ca623-0b27-4c15-ba21-5f3b6f0c7505-kube-api-access-xnwbz\") pod \"node-exporter-86n9r\" (UID: \"d72ca623-0b27-4c15-ba21-5f3b6f0c7505\") " pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.298121 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.298092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8f9\" (UniqueName: \"kubernetes.io/projected/1fa2f9ec-477d-4607-a9ae-492efc929777-kube-api-access-vz8f9\") pod \"openshift-state-metrics-67dcf8c979-5bdjz\" (UID: \"1fa2f9ec-477d-4607-a9ae-492efc929777\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.371080 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.371020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"
Sep 15 09:15:47.377187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.377172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"
Sep 15 09:15:47.396169 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.396147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86n9r"
Sep 15 09:15:47.410742 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:47.410711    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72ca623_0b27_4c15_ba21_5f3b6f0c7505.slice/crio-5b40e968658729300263f9cea26d997306cf3ee004ca05588d24a39cb3f10fd7 WatchSource:0}: Error finding container 5b40e968658729300263f9cea26d997306cf3ee004ca05588d24a39cb3f10fd7: Status 404 returned error can't find the container with id 5b40e968658729300263f9cea26d997306cf3ee004ca05588d24a39cb3f10fd7
Sep 15 09:15:47.646224 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.646154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj"]
Sep 15 09:15:47.649212 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:47.649173    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55de041_5230_4a5e_a871_ef60649a3c40.slice/crio-4bc60562f6a263aea13623665d5439d09b1ab8470940ce7219dd7c8ef22420cb WatchSource:0}: Error finding container 4bc60562f6a263aea13623665d5439d09b1ab8470940ce7219dd7c8ef22420cb: Status 404 returned error can't find the container with id 4bc60562f6a263aea13623665d5439d09b1ab8470940ce7219dd7c8ef22420cb
Sep 15 09:15:47.649388 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:47.649369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz"]
Sep 15 09:15:47.652751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:47.652730    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa2f9ec_477d_4607_a9ae_492efc929777.slice/crio-24fcd90acbdb8256f7070132cfaef269d8386d371f5fdd8ec1a08780fbd6d0ad WatchSource:0}: Error finding container 24fcd90acbdb8256f7070132cfaef269d8386d371f5fdd8ec1a08780fbd6d0ad: Status 404 returned error can't find the container with id 24fcd90acbdb8256f7070132cfaef269d8386d371f5fdd8ec1a08780fbd6d0ad
Sep 15 09:15:48.221377 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:48.221324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj" event={"ID":"e55de041-5230-4a5e-a871-ef60649a3c40","Type":"ContainerStarted","Data":"4bc60562f6a263aea13623665d5439d09b1ab8470940ce7219dd7c8ef22420cb"}
Sep 15 09:15:48.222911 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:48.222829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86n9r" event={"ID":"d72ca623-0b27-4c15-ba21-5f3b6f0c7505","Type":"ContainerStarted","Data":"5b40e968658729300263f9cea26d997306cf3ee004ca05588d24a39cb3f10fd7"}
Sep 15 09:15:48.225295 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:48.225267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz" event={"ID":"1fa2f9ec-477d-4607-a9ae-492efc929777","Type":"ContainerStarted","Data":"882be5deabbac2c006525b5890487731e005e9c97540972b68fd72831774bd71"}
Sep 15 09:15:48.225400 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:48.225328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz" event={"ID":"1fa2f9ec-477d-4607-a9ae-492efc929777","Type":"ContainerStarted","Data":"3452f059a02ae98ed10366d28c9741fc71d8ea2859b08da9deee5e04c4b674a8"}
Sep 15 09:15:48.225400 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:48.225380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz" event={"ID":"1fa2f9ec-477d-4607-a9ae-492efc929777","Type":"ContainerStarted","Data":"24fcd90acbdb8256f7070132cfaef269d8386d371f5fdd8ec1a08780fbd6d0ad"}
Sep 15 09:15:49.133732 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.133695    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-586844fb88-nhqkz"]
Sep 15 09:15:49.137816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.137791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.140378 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.140352    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 09:15:49.140378 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.140370    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-67s2bpshvkmhj\""
Sep 15 09:15:49.140686 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.140670    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 09:15:49.140757 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.140733    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 09:15:49.140866 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.140749    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jm4pr\""
Sep 15 09:15:49.140866 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.140735    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 09:15:49.141076 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.141058    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 09:15:49.147374 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.147347    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-586844fb88-nhqkz"]
Sep 15 09:15:49.230528 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.230497    2579 generic.go:356] "Generic (PLEG): container finished" podID="d72ca623-0b27-4c15-ba21-5f3b6f0c7505" containerID="59e41b43b28ca24583e214d2477bc1440ee802c766c43cb3fb3dc7189f9c4ee6" exitCode=0
Sep 15 09:15:49.230960 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.230532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86n9r" event={"ID":"d72ca623-0b27-4c15-ba21-5f3b6f0c7505","Type":"ContainerDied","Data":"59e41b43b28ca24583e214d2477bc1440ee802c766c43cb3fb3dc7189f9c4ee6"}
Sep 15 09:15:49.306770 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.306742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.306879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.306777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/321eb189-0735-4e62-9ddc-4975d8a3c1d9-metrics-client-ca\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.306879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.306799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-tls\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.306879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.306862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.307038 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.306904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.307038 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.306958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.307142 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.307054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-grpc-tls\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.307142 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.307086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbr6\" (UniqueName: \"kubernetes.io/projected/321eb189-0735-4e62-9ddc-4975d8a3c1d9-kube-api-access-wzbr6\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.408631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.407610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.408631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.407653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/321eb189-0735-4e62-9ddc-4975d8a3c1d9-metrics-client-ca\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.408631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.407704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-tls\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.408631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.407734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.408631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.407767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.408631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.407845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.408631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.407890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-grpc-tls\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.408631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.407930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbr6\" (UniqueName: \"kubernetes.io/projected/321eb189-0735-4e62-9ddc-4975d8a3c1d9-kube-api-access-wzbr6\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.410378 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.410144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/321eb189-0735-4e62-9ddc-4975d8a3c1d9-metrics-client-ca\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.411566 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.411527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.412200 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.411830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-tls\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.412200 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.412156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.413632 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.413592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-grpc-tls\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.413708 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.413649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.413875 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.413851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/321eb189-0735-4e62-9ddc-4975d8a3c1d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.417233 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.417215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbr6\" (UniqueName: \"kubernetes.io/projected/321eb189-0735-4e62-9ddc-4975d8a3c1d9-kube-api-access-wzbr6\") pod \"thanos-querier-586844fb88-nhqkz\" (UID: \"321eb189-0735-4e62-9ddc-4975d8a3c1d9\") " pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.449032 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.448826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:49.740803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:49.740770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-586844fb88-nhqkz"]
Sep 15 09:15:49.743944 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:49.743917    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321eb189_0735_4e62_9ddc_4975d8a3c1d9.slice/crio-a312045dc53ae00b461462a6b25dd0b00f367bf377ccac4f2c5de3f3d135c75f WatchSource:0}: Error finding container a312045dc53ae00b461462a6b25dd0b00f367bf377ccac4f2c5de3f3d135c75f: Status 404 returned error can't find the container with id a312045dc53ae00b461462a6b25dd0b00f367bf377ccac4f2c5de3f3d135c75f
Sep 15 09:15:50.236800 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.236759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj" event={"ID":"e55de041-5230-4a5e-a871-ef60649a3c40","Type":"ContainerStarted","Data":"1c7f3cdfab849ef5ad36a8c6cddccae7409ada7fc1fcc0d9b2145199976f2ca4"}
Sep 15 09:15:50.237242 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.236833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj" event={"ID":"e55de041-5230-4a5e-a871-ef60649a3c40","Type":"ContainerStarted","Data":"f0b2989bcbec2d48f5664dfe9a136f2af88f494f313ebb524f8db44fed8dbf06"}
Sep 15 09:15:50.237242 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.236867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj" event={"ID":"e55de041-5230-4a5e-a871-ef60649a3c40","Type":"ContainerStarted","Data":"e6bae0df05f3fcf8f158cf26919ba32a4f0a5a6cd6486f0046aabb7f509c0f99"}
Sep 15 09:15:50.238352 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.238297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz" event={"ID":"321eb189-0735-4e62-9ddc-4975d8a3c1d9","Type":"ContainerStarted","Data":"a312045dc53ae00b461462a6b25dd0b00f367bf377ccac4f2c5de3f3d135c75f"}
Sep 15 09:15:50.240955 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.240929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86n9r" event={"ID":"d72ca623-0b27-4c15-ba21-5f3b6f0c7505","Type":"ContainerStarted","Data":"303b1196851705e3be9be50716e069846dd1616fffb4483a479054b4a8b7df43"}
Sep 15 09:15:50.241055 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.240982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86n9r" event={"ID":"d72ca623-0b27-4c15-ba21-5f3b6f0c7505","Type":"ContainerStarted","Data":"3e5f02096603e33749f885486ab4cfec57a036151d0976efa57c76daa261f14a"}
Sep 15 09:15:50.243391 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.243365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz" event={"ID":"1fa2f9ec-477d-4607-a9ae-492efc929777","Type":"ContainerStarted","Data":"6fc22ee9e5f770cf6e7f3d2c66c68ff866cb783365da28d9d0c6364f09b7e525"}
Sep 15 09:15:50.257508 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.257467    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w2hdj" podStartSLOduration=1.637398107 podStartE2EDuration="3.257448182s" totalImagesPullingTime="1.620050075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:50.255422785 +0000 UTC m=+114.085214689" watchObservedRunningTime="2026-09-15 09:15:50.257448182 +0000 UTC m=+114.087240086"
Sep 15 09:15:50.272447 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.272406    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5bdjz" podStartSLOduration=1.7606958499999998 podStartE2EDuration="3.272393119s" totalImagesPullingTime="1.511697269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:50.272014549 +0000 UTC m=+114.101806452" watchObservedRunningTime="2026-09-15 09:15:50.272393119 +0000 UTC m=+114.102185023"
Sep 15 09:15:50.290360 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:50.290280    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-86n9r" podStartSLOduration=2.038779058 podStartE2EDuration="3.290262634s" totalImagesPullingTime="1.180253051s" totalInitContainerRuntime="71.230525ms" isStatefulPod=true podCreationTimestamp="2026-09-15 09:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:50.289829256 +0000 UTC m=+114.119621160" watchObservedRunningTime="2026-09-15 09:15:50.290262634 +0000 UTC m=+114.120054538"
Sep 15 09:15:51.359773 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.359736    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-747c78868f-bjv5v"]
Sep 15 09:15:51.363428 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.363406    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.365697 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.365677    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 09:15:51.365807 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.365737    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 09:15:51.365807 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.365738    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-l7nnx\""
Sep 15 09:15:51.366398 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.366377    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6k1vkp6b2ggtq\""
Sep 15 09:15:51.366398 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.366400    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 09:15:51.366575 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.366407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 09:15:51.374142 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.374122    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-747c78868f-bjv5v"]
Sep 15 09:15:51.526525 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.526483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98e2482a-d61f-4d97-a53e-ea27b1960c4d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.526708 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.526535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98e2482a-d61f-4d97-a53e-ea27b1960c4d-secret-metrics-server-tls\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.526708 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.526596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98e2482a-d61f-4d97-a53e-ea27b1960c4d-metrics-server-audit-profiles\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.526708 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.526623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98e2482a-d61f-4d97-a53e-ea27b1960c4d-client-ca-bundle\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.526708 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.526674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98e2482a-d61f-4d97-a53e-ea27b1960c4d-secret-metrics-server-client-certs\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.526708 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.526700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ttn\" (UniqueName: \"kubernetes.io/projected/98e2482a-d61f-4d97-a53e-ea27b1960c4d-kube-api-access-s8ttn\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.526901 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.526731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98e2482a-d61f-4d97-a53e-ea27b1960c4d-audit-log\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.627989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.627891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98e2482a-d61f-4d97-a53e-ea27b1960c4d-secret-metrics-server-client-certs\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.627989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.627946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ttn\" (UniqueName: \"kubernetes.io/projected/98e2482a-d61f-4d97-a53e-ea27b1960c4d-kube-api-access-s8ttn\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.627989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.627984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98e2482a-d61f-4d97-a53e-ea27b1960c4d-audit-log\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.628252 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.628053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98e2482a-d61f-4d97-a53e-ea27b1960c4d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.628252 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.628083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98e2482a-d61f-4d97-a53e-ea27b1960c4d-secret-metrics-server-tls\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.628252 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.628125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98e2482a-d61f-4d97-a53e-ea27b1960c4d-metrics-server-audit-profiles\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.628252 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.628151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98e2482a-d61f-4d97-a53e-ea27b1960c4d-client-ca-bundle\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.628649 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.628621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98e2482a-d61f-4d97-a53e-ea27b1960c4d-audit-log\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.629202 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.629150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98e2482a-d61f-4d97-a53e-ea27b1960c4d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.629369 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.629311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98e2482a-d61f-4d97-a53e-ea27b1960c4d-metrics-server-audit-profiles\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.631028 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.631001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98e2482a-d61f-4d97-a53e-ea27b1960c4d-secret-metrics-server-client-certs\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.631136 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.631063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98e2482a-d61f-4d97-a53e-ea27b1960c4d-secret-metrics-server-tls\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.631136 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.631101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98e2482a-d61f-4d97-a53e-ea27b1960c4d-client-ca-bundle\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.636269 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.636243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ttn\" (UniqueName: \"kubernetes.io/projected/98e2482a-d61f-4d97-a53e-ea27b1960c4d-kube-api-access-s8ttn\") pod \"metrics-server-747c78868f-bjv5v\" (UID: \"98e2482a-d61f-4d97-a53e-ea27b1960c4d\") " pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.674727 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.674689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:15:51.958411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:51.958369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-747c78868f-bjv5v"]
Sep 15 09:15:51.961428 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:51.961400    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e2482a_d61f_4d97_a53e_ea27b1960c4d.slice/crio-afdb09a2263ba6e6a317ac99a3aff5ad9dc25e535f45d3f5c3f0f9ffc2b47825 WatchSource:0}: Error finding container afdb09a2263ba6e6a317ac99a3aff5ad9dc25e535f45d3f5c3f0f9ffc2b47825: Status 404 returned error can't find the container with id afdb09a2263ba6e6a317ac99a3aff5ad9dc25e535f45d3f5c3f0f9ffc2b47825
Sep 15 09:15:52.253244 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:52.253216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz" event={"ID":"321eb189-0735-4e62-9ddc-4975d8a3c1d9","Type":"ContainerStarted","Data":"d45e24e92923c47da279135b3065475688ab2ec81ea8db9cbf1d2bd9dc918e94"}
Sep 15 09:15:52.253418 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:52.253291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz" event={"ID":"321eb189-0735-4e62-9ddc-4975d8a3c1d9","Type":"ContainerStarted","Data":"4371c28b0090b998769713edf6e3746f69e1b7e0bd2e129e2932a6f7e9038f2e"}
Sep 15 09:15:52.253418 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:52.253324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz" event={"ID":"321eb189-0735-4e62-9ddc-4975d8a3c1d9","Type":"ContainerStarted","Data":"1d4dd15783f0c598007b47b22c3f5175f3805cf4f66e44456a53b61174cf8463"}
Sep 15 09:15:52.254387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:52.254368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-747c78868f-bjv5v" event={"ID":"98e2482a-d61f-4d97-a53e-ea27b1960c4d","Type":"ContainerStarted","Data":"afdb09a2263ba6e6a317ac99a3aff5ad9dc25e535f45d3f5c3f0f9ffc2b47825"}
Sep 15 09:15:53.365921 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.365754    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:15:53.370023 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.369988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.374533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.374480    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 09:15:53.374533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.374482    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8t70g6gjkfqi3\""
Sep 15 09:15:53.374971 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.374947    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 09:15:53.374971 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.374971    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 09:15:53.375590 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.375568    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2pkjg\""
Sep 15 09:15:53.375590 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.375581    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 09:15:53.375770 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.375564    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 09:15:53.375770 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.375729    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 09:15:53.375994 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.375974    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 09:15:53.376067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.376040    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 09:15:53.376127 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.376111    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 09:15:53.376648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.376606    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 09:15:53.376772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.376655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 09:15:53.376772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.376675    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 09:15:53.377417 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.377395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 09:15:53.378130 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.378109    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 09:15:53.387975 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.387936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:15:53.544696 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl69t\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-kube-api-access-vl69t\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.544864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.544864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.544864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.544864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.544983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-config-out\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-web-config\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545583 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.545583 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.545379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-config\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-web-config\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-config\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl69t\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-kube-api-access-vl69t\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-config-out\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.647187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.648138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.648138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.648138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.646930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.648138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.647577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.648138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.647857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.648138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.647857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.648505 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.648476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.651299 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.650278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.652256 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.652229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.652987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.652957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.653756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.653730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.654240 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.654219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.654411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.654246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-config\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.654539 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.654423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-web-config\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.654779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.654754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.659457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.658617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.660189 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.660135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.660273 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.660261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.660919 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.660890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.661019 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.660987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.661746 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.661714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-config-out\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.662248 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.662225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl69t\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-kube-api-access-vl69t\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.662319 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.662254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.687510 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.687304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:15:53.978709 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:53.978678    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:15:53.981951 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:15:53.981915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c80b0a_5b94_478b_a74d_0278759e6741.slice/crio-d5ee0abd7f6580d852d463f8b43a78d74ea5929dd40644eddbef56c4e82f147a WatchSource:0}: Error finding container d5ee0abd7f6580d852d463f8b43a78d74ea5929dd40644eddbef56c4e82f147a: Status 404 returned error can't find the container with id d5ee0abd7f6580d852d463f8b43a78d74ea5929dd40644eddbef56c4e82f147a
Sep 15 09:15:54.264463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:54.264369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerStarted","Data":"d5ee0abd7f6580d852d463f8b43a78d74ea5929dd40644eddbef56c4e82f147a"}
Sep 15 09:15:54.267456 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:54.267421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz" event={"ID":"321eb189-0735-4e62-9ddc-4975d8a3c1d9","Type":"ContainerStarted","Data":"aec605cc383fedbfa8dcbab84110a8c6da8d0f9036c562ca71cfa02650c17464"}
Sep 15 09:15:54.267647 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:54.267491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz" event={"ID":"321eb189-0735-4e62-9ddc-4975d8a3c1d9","Type":"ContainerStarted","Data":"fb90d1b6c45208f0045fac96360c300e031fcb31e961f967df70d550a09e13f4"}
Sep 15 09:15:54.267647 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:54.267519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz" event={"ID":"321eb189-0735-4e62-9ddc-4975d8a3c1d9","Type":"ContainerStarted","Data":"63f2a133b12ab9ad9430bbde4d04268ba8570129440baa90001008691c50e086"}
Sep 15 09:15:54.267764 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:54.267661    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:15:54.269159 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:54.269132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-747c78868f-bjv5v" event={"ID":"98e2482a-d61f-4d97-a53e-ea27b1960c4d","Type":"ContainerStarted","Data":"405296230bc6de4a111a421c5f1698509ca5d7fb5c5955b602166937e03736d3"}
Sep 15 09:15:54.290772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:54.290703    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz" podStartSLOduration=1.547928244 podStartE2EDuration="5.290677974s" totalImagesPullingTime="3.74274973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:15:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:54.289127328 +0000 UTC m=+118.118919232" watchObservedRunningTime="2026-09-15 09:15:54.290677974 +0000 UTC m=+118.120469879"
Sep 15 09:15:54.305422 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:54.305363    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-747c78868f-bjv5v" podStartSLOduration=1.583213245 podStartE2EDuration="3.305326373s" totalImagesPullingTime="1.722113128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:54.304872882 +0000 UTC m=+118.134664786" watchObservedRunningTime="2026-09-15 09:15:54.305326373 +0000 UTC m=+118.135118276"
Sep 15 09:15:55.274444 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:55.274412    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerID="8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8" exitCode=0
Sep 15 09:15:55.274860 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:55.274511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerDied","Data":"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"}
Sep 15 09:15:59.295828 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:59.295740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerStarted","Data":"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"}
Sep 15 09:15:59.295828 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:59.295805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerStarted","Data":"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"}
Sep 15 09:15:59.296229 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:59.295830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerStarted","Data":"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"}
Sep 15 09:15:59.296229 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:59.295857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerStarted","Data":"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"}
Sep 15 09:15:59.296229 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:59.295886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerStarted","Data":"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"}
Sep 15 09:15:59.296229 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:59.295933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerStarted","Data":"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"}
Sep 15 09:15:59.334163 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:15:59.334104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.8153140429999999 podStartE2EDuration="6.334082597s" totalImagesPullingTime="4.49535857s" totalInitContainerRuntime="23.409984ms" isStatefulPod=false podCreationTimestamp="2026-09-15 09:15:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:59.332833853 +0000 UTC m=+123.162625756" watchObservedRunningTime="2026-09-15 09:15:59.334082597 +0000 UTC m=+123.163874513"
Sep 15 09:16:00.280643 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:00.280613    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-586844fb88-nhqkz"
Sep 15 09:16:03.688635 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:03.688577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:16:06.463151 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:06.463100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:16:06.465669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:06.465640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26f8b6a1-afde-4839-9886-05fc1b9f1159-metrics-certs\") pod \"network-metrics-daemon-kzmvm\" (UID: \"26f8b6a1-afde-4839-9886-05fc1b9f1159\") " pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:16:06.694460 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:06.694426    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ddxwx\""
Sep 15 09:16:06.703394 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:06.703366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzmvm"
Sep 15 09:16:06.975660 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:06.975634    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kzmvm"]
Sep 15 09:16:06.978170 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:16:06.978131    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f8b6a1_afde_4839_9886_05fc1b9f1159.slice/crio-9a1eed104764a53f8ec56480acd96549a1c35b1f42b9b579eb7b8211aee63230 WatchSource:0}: Error finding container 9a1eed104764a53f8ec56480acd96549a1c35b1f42b9b579eb7b8211aee63230: Status 404 returned error can't find the container with id 9a1eed104764a53f8ec56480acd96549a1c35b1f42b9b579eb7b8211aee63230
Sep 15 09:16:07.329382 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:07.329270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzmvm" event={"ID":"26f8b6a1-afde-4839-9886-05fc1b9f1159","Type":"ContainerStarted","Data":"9a1eed104764a53f8ec56480acd96549a1c35b1f42b9b579eb7b8211aee63230"}
Sep 15 09:16:08.334920 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:08.334881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzmvm" event={"ID":"26f8b6a1-afde-4839-9886-05fc1b9f1159","Type":"ContainerStarted","Data":"e8f2cf7cfc42066d40eba0ab27d1749229f1bd1107bae8ea0a8e60040d3e66ae"}
Sep 15 09:16:08.335307 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:08.334933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzmvm" event={"ID":"26f8b6a1-afde-4839-9886-05fc1b9f1159","Type":"ContainerStarted","Data":"c7748790c9707dbedd06b556c036a3d8ea1a642505bcbb389e1e29319f118eda"}
Sep 15 09:16:11.675495 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:11.675457    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:16:11.675869 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:11.675537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:16:31.683030 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:31.682995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:16:31.686996 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:31.686971    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-747c78868f-bjv5v"
Sep 15 09:16:31.704571 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:31.704517    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kzmvm" podStartSLOduration=154.586197755 podStartE2EDuration="2m35.704494691s" totalImagesPullingTime="1.118296936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:16:08.349083007 +0000 UTC m=+132.178874913" watchObservedRunningTime="2026-09-15 09:16:31.704494691 +0000 UTC m=+155.534286595"
Sep 15 09:16:53.688278 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:53.688240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:16:53.706771 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:53.706748    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:16:54.533506 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:16:54.533479    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:11.769516 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:11.769484    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:17:11.770091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:11.769835    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="prometheus" containerID="cri-o://bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4" gracePeriod=600
Sep 15 09:17:11.770091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:11.769874    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy" containerID="cri-o://f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d" gracePeriod=600
Sep 15 09:17:11.770091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:11.769922    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="thanos-sidecar" containerID="cri-o://c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7" gracePeriod=600
Sep 15 09:17:11.770091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:11.769949    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy-thanos" containerID="cri-o://820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e" gracePeriod=600
Sep 15 09:17:11.770091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:11.769939    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy-web" containerID="cri-o://fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c" gracePeriod=600
Sep 15 09:17:11.770379 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:11.770098    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="config-reloader" containerID="cri-o://9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6" gracePeriod=600
Sep 15 09:17:12.017350 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.015931    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.103037 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.102951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-web-config\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103037 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.102986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-serving-certs-ca-bundle\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103037 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-metrics-client-ca\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-config-out\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103072    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-metrics-client-certs\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-2\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-0\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-tls\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-config\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103226    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-tls-assets\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103254    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-kube-rbac-proxy\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103284    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-thanos-prometheus-http-client-file\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103313    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl69t\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-kube-api-access-vl69t\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-kubelet-serving-ca-bundle\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103331    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103386    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-1\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103454    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103695    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103782    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-grpc-tls\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-db\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.103888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.103842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-trusted-ca-bundle\") pod \"f5c80b0a-5b94-478b-a74d-0278759e6741\" (UID: \"f5c80b0a-5b94-478b-a74d-0278759e6741\") "
Sep 15 09:17:12.104469 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.104106    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-metrics-client-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.104469 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.104126    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.104469 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.104141    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.104469 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.104160    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.104983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.104923    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:17:12.105687 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.105659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:17:12.106050 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.106018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-config-out" (OuterVolumeSpecName: "config-out") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:17:12.107012 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.106972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:17:12.107525 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.107180    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:17:12.107525 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.107281    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:17:12.108496 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.107864    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:17:12.108496 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.107881    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:17:12.108788 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.108696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:17:12.108788 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.108766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:17:12.108992 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.108972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-kube-api-access-vl69t" (OuterVolumeSpecName: "kube-api-access-vl69t") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "kube-api-access-vl69t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:17:12.109055 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.109030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:17:12.109265 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.109223    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:17:12.109379 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.109308    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:17:12.109883 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.109863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-config" (OuterVolumeSpecName: "config") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:17:12.117131 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.117109    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-web-config" (OuterVolumeSpecName: "web-config") pod "f5c80b0a-5b94-478b-a74d-0278759e6741" (UID: "f5c80b0a-5b94-478b-a74d-0278759e6741"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:17:12.204796 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204764    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-config\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204796 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204795    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-tls-assets\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204810    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-kube-rbac-proxy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204820    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204830    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl69t\" (UniqueName: \"kubernetes.io/projected/f5c80b0a-5b94-478b-a74d-0278759e6741-kube-api-access-vl69t\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204839    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204848    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204857    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204867    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-grpc-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204875    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-db\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204883    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204892    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-web-config\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204899    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c80b0a-5b94-478b-a74d-0278759e6741-config-out\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204907    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-metrics-client-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204917    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5c80b0a-5b94-478b-a74d-0278759e6741-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.204944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.204931    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5c80b0a-5b94-478b-a74d-0278759e6741-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:17:12.591894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.591863    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerID="820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e" exitCode=0
Sep 15 09:17:12.591894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.591888    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerID="f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d" exitCode=0
Sep 15 09:17:12.591894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.591896    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerID="fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c" exitCode=0
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.591906    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerID="c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7" exitCode=0
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.591915    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerID="9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6" exitCode=0
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.591924    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerID="bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4" exitCode=0
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.591941    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.591938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerDied","Data":"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"}
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.592064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerDied","Data":"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"}
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.592091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerDied","Data":"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"}
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.592115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerDied","Data":"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"}
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.592135    2579 scope.go:117] "RemoveContainer" containerID="820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"
Sep 15 09:17:12.592153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.592141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerDied","Data":"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"}
Sep 15 09:17:12.592589 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.592166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerDied","Data":"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"}
Sep 15 09:17:12.592589 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.592193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5c80b0a-5b94-478b-a74d-0278759e6741","Type":"ContainerDied","Data":"d5ee0abd7f6580d852d463f8b43a78d74ea5929dd40644eddbef56c4e82f147a"}
Sep 15 09:17:12.601867 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.601845    2579 scope.go:117] "RemoveContainer" containerID="f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"
Sep 15 09:17:12.609454 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.609437    2579 scope.go:117] "RemoveContainer" containerID="fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"
Sep 15 09:17:12.615715 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.615699    2579 scope.go:117] "RemoveContainer" containerID="c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"
Sep 15 09:17:12.620258 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.620242    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:17:12.622898 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.622879    2579 scope.go:117] "RemoveContainer" containerID="9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"
Sep 15 09:17:12.628997 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.628977    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:17:12.630135 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.630122    2579 scope.go:117] "RemoveContainer" containerID="bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"
Sep 15 09:17:12.636714 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.636694    2579 scope.go:117] "RemoveContainer" containerID="8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"
Sep 15 09:17:12.642896 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.642877    2579 scope.go:117] "RemoveContainer" containerID="820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"
Sep 15 09:17:12.643149 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:17:12.643132    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": container with ID starting with 820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e not found: ID does not exist" containerID="820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"
Sep 15 09:17:12.643201 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.643156    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"} err="failed to get container status \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": rpc error: code = NotFound desc = could not find container \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": container with ID starting with 820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e not found: ID does not exist"
Sep 15 09:17:12.643201 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.643173    2579 scope.go:117] "RemoveContainer" containerID="f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"
Sep 15 09:17:12.643449 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:17:12.643431    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": container with ID starting with f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d not found: ID does not exist" containerID="f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"
Sep 15 09:17:12.643497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.643459    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"} err="failed to get container status \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": rpc error: code = NotFound desc = could not find container \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": container with ID starting with f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d not found: ID does not exist"
Sep 15 09:17:12.643497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.643474    2579 scope.go:117] "RemoveContainer" containerID="fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"
Sep 15 09:17:12.643729 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:17:12.643712    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": container with ID starting with fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c not found: ID does not exist" containerID="fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"
Sep 15 09:17:12.643777 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.643734    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"} err="failed to get container status \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": rpc error: code = NotFound desc = could not find container \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": container with ID starting with fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c not found: ID does not exist"
Sep 15 09:17:12.643777 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.643757    2579 scope.go:117] "RemoveContainer" containerID="c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"
Sep 15 09:17:12.643996 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:17:12.643980    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": container with ID starting with c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7 not found: ID does not exist" containerID="c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"
Sep 15 09:17:12.644047 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.643999    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"} err="failed to get container status \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": rpc error: code = NotFound desc = could not find container \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": container with ID starting with c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7 not found: ID does not exist"
Sep 15 09:17:12.644047 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.644014    2579 scope.go:117] "RemoveContainer" containerID="9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"
Sep 15 09:17:12.644235 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:17:12.644219    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": container with ID starting with 9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6 not found: ID does not exist" containerID="9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"
Sep 15 09:17:12.644273 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.644239    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"} err="failed to get container status \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": rpc error: code = NotFound desc = could not find container \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": container with ID starting with 9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6 not found: ID does not exist"
Sep 15 09:17:12.644273 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.644254    2579 scope.go:117] "RemoveContainer" containerID="bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"
Sep 15 09:17:12.644535 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:17:12.644506    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": container with ID starting with bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4 not found: ID does not exist" containerID="bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"
Sep 15 09:17:12.644535 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.644529    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"} err="failed to get container status \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": rpc error: code = NotFound desc = could not find container \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": container with ID starting with bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4 not found: ID does not exist"
Sep 15 09:17:12.644683 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.644544    2579 scope.go:117] "RemoveContainer" containerID="8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"
Sep 15 09:17:12.644782 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:17:12.644763    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": container with ID starting with 8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8 not found: ID does not exist" containerID="8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"
Sep 15 09:17:12.644875 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.644782    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"} err="failed to get container status \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": rpc error: code = NotFound desc = could not find container \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": container with ID starting with 8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8 not found: ID does not exist"
Sep 15 09:17:12.644875 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.644793    2579 scope.go:117] "RemoveContainer" containerID="820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"
Sep 15 09:17:12.645025 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645006    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"} err="failed to get container status \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": rpc error: code = NotFound desc = could not find container \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": container with ID starting with 820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e not found: ID does not exist"
Sep 15 09:17:12.645074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645027    2579 scope.go:117] "RemoveContainer" containerID="f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"
Sep 15 09:17:12.645258 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645240    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"} err="failed to get container status \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": rpc error: code = NotFound desc = could not find container \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": container with ID starting with f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d not found: ID does not exist"
Sep 15 09:17:12.645318 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645259    2579 scope.go:117] "RemoveContainer" containerID="fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"
Sep 15 09:17:12.645481 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645466    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"} err="failed to get container status \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": rpc error: code = NotFound desc = could not find container \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": container with ID starting with fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c not found: ID does not exist"
Sep 15 09:17:12.645524 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645481    2579 scope.go:117] "RemoveContainer" containerID="c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"
Sep 15 09:17:12.645668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645651    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"} err="failed to get container status \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": rpc error: code = NotFound desc = could not find container \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": container with ID starting with c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7 not found: ID does not exist"
Sep 15 09:17:12.645707 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645668    2579 scope.go:117] "RemoveContainer" containerID="9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"
Sep 15 09:17:12.645877 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645861    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"} err="failed to get container status \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": rpc error: code = NotFound desc = could not find container \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": container with ID starting with 9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6 not found: ID does not exist"
Sep 15 09:17:12.645916 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.645878    2579 scope.go:117] "RemoveContainer" containerID="bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"
Sep 15 09:17:12.646069 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646055    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"} err="failed to get container status \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": rpc error: code = NotFound desc = could not find container \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": container with ID starting with bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4 not found: ID does not exist"
Sep 15 09:17:12.646110 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646070    2579 scope.go:117] "RemoveContainer" containerID="8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"
Sep 15 09:17:12.646272 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646254    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"} err="failed to get container status \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": rpc error: code = NotFound desc = could not find container \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": container with ID starting with 8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8 not found: ID does not exist"
Sep 15 09:17:12.646315 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646272    2579 scope.go:117] "RemoveContainer" containerID="820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"
Sep 15 09:17:12.646490 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646475    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"} err="failed to get container status \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": rpc error: code = NotFound desc = could not find container \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": container with ID starting with 820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e not found: ID does not exist"
Sep 15 09:17:12.646538 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646490    2579 scope.go:117] "RemoveContainer" containerID="f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"
Sep 15 09:17:12.646700 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646684    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"} err="failed to get container status \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": rpc error: code = NotFound desc = could not find container \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": container with ID starting with f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d not found: ID does not exist"
Sep 15 09:17:12.646742 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646700    2579 scope.go:117] "RemoveContainer" containerID="fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"
Sep 15 09:17:12.646899 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646881    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"} err="failed to get container status \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": rpc error: code = NotFound desc = could not find container \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": container with ID starting with fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c not found: ID does not exist"
Sep 15 09:17:12.646944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.646899    2579 scope.go:117] "RemoveContainer" containerID="c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"
Sep 15 09:17:12.647093 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647073    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"} err="failed to get container status \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": rpc error: code = NotFound desc = could not find container \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": container with ID starting with c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7 not found: ID does not exist"
Sep 15 09:17:12.647163 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647096    2579 scope.go:117] "RemoveContainer" containerID="9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"
Sep 15 09:17:12.647302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647284    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"} err="failed to get container status \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": rpc error: code = NotFound desc = could not find container \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": container with ID starting with 9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6 not found: ID does not exist"
Sep 15 09:17:12.647407 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647303    2579 scope.go:117] "RemoveContainer" containerID="bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"
Sep 15 09:17:12.647514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647498    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"} err="failed to get container status \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": rpc error: code = NotFound desc = could not find container \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": container with ID starting with bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4 not found: ID does not exist"
Sep 15 09:17:12.647551 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647513    2579 scope.go:117] "RemoveContainer" containerID="8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"
Sep 15 09:17:12.647725 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647710    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"} err="failed to get container status \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": rpc error: code = NotFound desc = could not find container \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": container with ID starting with 8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8 not found: ID does not exist"
Sep 15 09:17:12.647790 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647726    2579 scope.go:117] "RemoveContainer" containerID="820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"
Sep 15 09:17:12.647928 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647910    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"} err="failed to get container status \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": rpc error: code = NotFound desc = could not find container \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": container with ID starting with 820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e not found: ID does not exist"
Sep 15 09:17:12.647981 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.647927    2579 scope.go:117] "RemoveContainer" containerID="f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"
Sep 15 09:17:12.648202 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.648151    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"} err="failed to get container status \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": rpc error: code = NotFound desc = could not find container \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": container with ID starting with f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d not found: ID does not exist"
Sep 15 09:17:12.648202 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.648200    2579 scope.go:117] "RemoveContainer" containerID="fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"
Sep 15 09:17:12.648482 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.648462    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"} err="failed to get container status \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": rpc error: code = NotFound desc = could not find container \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": container with ID starting with fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c not found: ID does not exist"
Sep 15 09:17:12.648482 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.648481    2579 scope.go:117] "RemoveContainer" containerID="c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"
Sep 15 09:17:12.648736 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.648710    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"} err="failed to get container status \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": rpc error: code = NotFound desc = could not find container \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": container with ID starting with c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7 not found: ID does not exist"
Sep 15 09:17:12.648810 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.648736    2579 scope.go:117] "RemoveContainer" containerID="9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"
Sep 15 09:17:12.649000 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.648978    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"} err="failed to get container status \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": rpc error: code = NotFound desc = could not find container \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": container with ID starting with 9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6 not found: ID does not exist"
Sep 15 09:17:12.649000 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.648998    2579 scope.go:117] "RemoveContainer" containerID="bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"
Sep 15 09:17:12.649326 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.649307    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"} err="failed to get container status \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": rpc error: code = NotFound desc = could not find container \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": container with ID starting with bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4 not found: ID does not exist"
Sep 15 09:17:12.649413 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.649328    2579 scope.go:117] "RemoveContainer" containerID="8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"
Sep 15 09:17:12.649591 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.649575    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"} err="failed to get container status \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": rpc error: code = NotFound desc = could not find container \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": container with ID starting with 8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8 not found: ID does not exist"
Sep 15 09:17:12.649643 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.649591    2579 scope.go:117] "RemoveContainer" containerID="820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"
Sep 15 09:17:12.649819 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.649802    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"} err="failed to get container status \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": rpc error: code = NotFound desc = could not find container \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": container with ID starting with 820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e not found: ID does not exist"
Sep 15 09:17:12.649859 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.649821    2579 scope.go:117] "RemoveContainer" containerID="f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"
Sep 15 09:17:12.650046 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650022    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"} err="failed to get container status \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": rpc error: code = NotFound desc = could not find container \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": container with ID starting with f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d not found: ID does not exist"
Sep 15 09:17:12.650126 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650044    2579 scope.go:117] "RemoveContainer" containerID="fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"
Sep 15 09:17:12.650300 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650283    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"} err="failed to get container status \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": rpc error: code = NotFound desc = could not find container \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": container with ID starting with fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c not found: ID does not exist"
Sep 15 09:17:12.650385 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650302    2579 scope.go:117] "RemoveContainer" containerID="c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"
Sep 15 09:17:12.650566 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650547    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"} err="failed to get container status \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": rpc error: code = NotFound desc = could not find container \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": container with ID starting with c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7 not found: ID does not exist"
Sep 15 09:17:12.650610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650567    2579 scope.go:117] "RemoveContainer" containerID="9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"
Sep 15 09:17:12.650769 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650755    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"} err="failed to get container status \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": rpc error: code = NotFound desc = could not find container \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": container with ID starting with 9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6 not found: ID does not exist"
Sep 15 09:17:12.650769 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650769    2579 scope.go:117] "RemoveContainer" containerID="bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"
Sep 15 09:17:12.650978 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650961    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"} err="failed to get container status \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": rpc error: code = NotFound desc = could not find container \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": container with ID starting with bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4 not found: ID does not exist"
Sep 15 09:17:12.651022 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.650979    2579 scope.go:117] "RemoveContainer" containerID="8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"
Sep 15 09:17:12.651230 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.651213    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"} err="failed to get container status \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": rpc error: code = NotFound desc = could not find container \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": container with ID starting with 8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8 not found: ID does not exist"
Sep 15 09:17:12.651299 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.651234    2579 scope.go:117] "RemoveContainer" containerID="820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"
Sep 15 09:17:12.651460 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.651442    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e"} err="failed to get container status \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": rpc error: code = NotFound desc = could not find container \"820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e\": container with ID starting with 820ea4de802ff325cfc11d988733629c617e932bc6bc67cdf76dcf326105619e not found: ID does not exist"
Sep 15 09:17:12.651521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.651461    2579 scope.go:117] "RemoveContainer" containerID="f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"
Sep 15 09:17:12.651655 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.651638    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d"} err="failed to get container status \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": rpc error: code = NotFound desc = could not find container \"f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d\": container with ID starting with f213ce148f4ba3b50222235f5831499c1151629024ac6389f1dee136a16a7f2d not found: ID does not exist"
Sep 15 09:17:12.651713 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.651655    2579 scope.go:117] "RemoveContainer" containerID="fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"
Sep 15 09:17:12.651882 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.651858    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c"} err="failed to get container status \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": rpc error: code = NotFound desc = could not find container \"fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c\": container with ID starting with fad544947acbc631085bb38437576809b32cdb362a1a6b2d316e0d4e08ba3e5c not found: ID does not exist"
Sep 15 09:17:12.651956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.651884    2579 scope.go:117] "RemoveContainer" containerID="c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"
Sep 15 09:17:12.652094 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.652079    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7"} err="failed to get container status \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": rpc error: code = NotFound desc = could not find container \"c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7\": container with ID starting with c898e4ddabeeaaa560be5a5571b5707ef222e5f49e6725434a4391b3989fe6c7 not found: ID does not exist"
Sep 15 09:17:12.652142 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.652094    2579 scope.go:117] "RemoveContainer" containerID="9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"
Sep 15 09:17:12.652319 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.652298    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6"} err="failed to get container status \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": rpc error: code = NotFound desc = could not find container \"9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6\": container with ID starting with 9457a77a9f838c81e68049dbe9da47e1b97b08b9dbd3d5dbbdd40fcd7d2eb5d6 not found: ID does not exist"
Sep 15 09:17:12.652432 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.652321    2579 scope.go:117] "RemoveContainer" containerID="bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"
Sep 15 09:17:12.652593 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.652575    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4"} err="failed to get container status \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": rpc error: code = NotFound desc = could not find container \"bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4\": container with ID starting with bdc1ee6a56645e97f678de24027c32483c0f397e3c4720907f6c2cbba5f55ab4 not found: ID does not exist"
Sep 15 09:17:12.652643 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.652592    2579 scope.go:117] "RemoveContainer" containerID="8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"
Sep 15 09:17:12.652791 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.652774    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8"} err="failed to get container status \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": rpc error: code = NotFound desc = could not find container \"8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8\": container with ID starting with 8f005be49e33289b372b8659cd3d3c0031c5a8d38e67d663a6d22c996d2a28d8 not found: ID does not exist"
Sep 15 09:17:12.657918 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.657899    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:17:12.658162 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658151    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy-web"
Sep 15 09:17:12.658203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658163    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy-web"
Sep 15 09:17:12.658203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658172    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy"
Sep 15 09:17:12.658203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658177    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy"
Sep 15 09:17:12.658203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658186    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="prometheus"
Sep 15 09:17:12.658203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658191    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="prometheus"
Sep 15 09:17:12.658203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658199    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="init-config-reloader"
Sep 15 09:17:12.658203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658204    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="init-config-reloader"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658210    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="config-reloader"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658216    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="config-reloader"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658229    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="thanos-sidecar"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658234    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="thanos-sidecar"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658239    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy-thanos"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658244    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy-thanos"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658287    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy-web"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658295    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy-thanos"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658301    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="prometheus"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658306    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="kube-rbac-proxy"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658312    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="config-reloader"
Sep 15 09:17:12.658463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.658316    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" containerName="thanos-sidecar"
Sep 15 09:17:12.663108 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.663094    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.666639 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.666621    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 09:17:12.666736 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.666674    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 09:17:12.666736 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.666683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 09:17:12.666961 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.666946    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 09:17:12.666961 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.666959    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 09:17:12.667110 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.667098    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 09:17:12.667331 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.667315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 09:17:12.667331 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.667320    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8t70g6gjkfqi3\""
Sep 15 09:17:12.667481 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.667414    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 09:17:12.667481 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.667416    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 09:17:12.667577 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.667476    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 09:17:12.667738 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.667714    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 09:17:12.667831 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.667772    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2pkjg\""
Sep 15 09:17:12.667831 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.667717    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 09:17:12.670392 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.670250    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 09:17:12.678323 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.678299    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 09:17:12.679277 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.679259    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5c80b0a-5b94-478b-a74d-0278759e6741" path="/var/lib/kubelet/pods/f5c80b0a-5b94-478b-a74d-0278759e6741/volumes"
Sep 15 09:17:12.683084 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.683064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:17:12.809235 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-config\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-config-out\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.809600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.810009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfg4c\" (UniqueName: \"kubernetes.io/projected/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-kube-api-access-kfg4c\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.810009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.810009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.810009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.810009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-web-config\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.810009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.810009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.810009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.810009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.809925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfg4c\" (UniqueName: \"kubernetes.io/projected/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-kube-api-access-kfg4c\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910617 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910617 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910617 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910617 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-web-config\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910617 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910834 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910834 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910834 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910834 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910834 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.910834 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-config\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-config-out\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.910978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.911007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.911032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.911361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.911373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.911627 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.911569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.913373 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.913327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.913464 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.913444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.914110 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.914080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.914270 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.914224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.914756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.914399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.914756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.914489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-web-config\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.914756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.914705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-config\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.914756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.914713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.915220 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.915197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.915855 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.915509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.916084 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.916062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.916165 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.916070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.916313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.916287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.916464 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.916437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-config-out\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.916549 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.916537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.916820 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.916800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.918639 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.918617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfg4c\" (UniqueName: \"kubernetes.io/projected/71a8d178-6e82-498d-8929-1bd0f4e3ddb8-kube-api-access-kfg4c\") pod \"prometheus-k8s-0\" (UID: \"71a8d178-6e82-498d-8929-1bd0f4e3ddb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:12.974652 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:12.974631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:17:13.257903 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:13.257878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:17:13.260233 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:17:13.260209    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a8d178_6e82_498d_8929_1bd0f4e3ddb8.slice/crio-ce9fd2f1fe335679ca485b34b7ac80228d905f9e88279eff1a547180931cab23 WatchSource:0}: Error finding container ce9fd2f1fe335679ca485b34b7ac80228d905f9e88279eff1a547180931cab23: Status 404 returned error can't find the container with id ce9fd2f1fe335679ca485b34b7ac80228d905f9e88279eff1a547180931cab23
Sep 15 09:17:13.597278 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:13.597182    2579 generic.go:356] "Generic (PLEG): container finished" podID="71a8d178-6e82-498d-8929-1bd0f4e3ddb8" containerID="17e4d7d886a7d3ac2b3e5b12269c739239c20a5bbcd9f99fa5ddbbd8896affff" exitCode=0
Sep 15 09:17:13.597463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:13.597275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71a8d178-6e82-498d-8929-1bd0f4e3ddb8","Type":"ContainerDied","Data":"17e4d7d886a7d3ac2b3e5b12269c739239c20a5bbcd9f99fa5ddbbd8896affff"}
Sep 15 09:17:13.597463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:13.597391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71a8d178-6e82-498d-8929-1bd0f4e3ddb8","Type":"ContainerStarted","Data":"ce9fd2f1fe335679ca485b34b7ac80228d905f9e88279eff1a547180931cab23"}
Sep 15 09:17:14.605934 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:14.605894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71a8d178-6e82-498d-8929-1bd0f4e3ddb8","Type":"ContainerStarted","Data":"f54ab871389a777ac79a086523fe306bd8025c6e8944b85da6be34275bd2c783"}
Sep 15 09:17:14.606322 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:14.605984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71a8d178-6e82-498d-8929-1bd0f4e3ddb8","Type":"ContainerStarted","Data":"88e5a00083b49d5b2e33504ab706c1bfb624e5a7c5ee7bb562d2f1303b5435c1"}
Sep 15 09:17:14.606322 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:14.606023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71a8d178-6e82-498d-8929-1bd0f4e3ddb8","Type":"ContainerStarted","Data":"580a5e8e5303652e8b609d28fe4c5fc427180eeba9c12b3c53cc1fb98930b300"}
Sep 15 09:17:14.606322 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:14.606063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71a8d178-6e82-498d-8929-1bd0f4e3ddb8","Type":"ContainerStarted","Data":"907a92444d8adba3eb35b3e586f3022afd1edd0715e5a5e462e00a2b052aee7b"}
Sep 15 09:17:14.606322 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:14.606105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71a8d178-6e82-498d-8929-1bd0f4e3ddb8","Type":"ContainerStarted","Data":"9f6c80e05a2d1f8cb3ca12bce8322034b8ed79533a411e6e81f5b7a350fd4aa3"}
Sep 15 09:17:14.606322 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:14.606142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71a8d178-6e82-498d-8929-1bd0f4e3ddb8","Type":"ContainerStarted","Data":"21534872a3cb5dac7e062d07f5b18e29adc25fc3d238983c6bff30c8ebafb562"}
Sep 15 09:17:14.637249 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:14.637182    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.616095878 podStartE2EDuration="2.637155015s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.059137ms" isStatefulPod=false podCreationTimestamp="2026-09-15 09:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:17:14.635468539 +0000 UTC m=+198.465260443" watchObservedRunningTime="2026-09-15 09:17:14.637155015 +0000 UTC m=+198.466946919"
Sep 15 09:17:17.974905 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:17:17.974867    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:18:12.974755 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:18:12.974723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:18:12.988574 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:18:12.988549    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:18:13.841662 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:18:13.841635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:18:56.590421 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:18:56.590391    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 09:20:06.830869 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:06.830839    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tssd9"]
Sep 15 09:20:06.833782 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:06.833764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:06.835929 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:06.835903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vktxr\""
Sep 15 09:20:06.836029 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:06.835912    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 09:20:06.836556 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:06.836540    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 09:20:06.841866 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:06.841842    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tssd9"]
Sep 15 09:20:06.977548 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:06.977509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dac8a5cc-34f6-4f86-8a4c-9f8273878bc4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tssd9\" (UID: \"dac8a5cc-34f6-4f86-8a4c-9f8273878bc4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:06.977701 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:06.977569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcjq7\" (UniqueName: \"kubernetes.io/projected/dac8a5cc-34f6-4f86-8a4c-9f8273878bc4-kube-api-access-jcjq7\") pod \"cert-manager-webhook-6486ffd997-tssd9\" (UID: \"dac8a5cc-34f6-4f86-8a4c-9f8273878bc4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:07.078315 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:07.078281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dac8a5cc-34f6-4f86-8a4c-9f8273878bc4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tssd9\" (UID: \"dac8a5cc-34f6-4f86-8a4c-9f8273878bc4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:07.078476 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:07.078353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcjq7\" (UniqueName: \"kubernetes.io/projected/dac8a5cc-34f6-4f86-8a4c-9f8273878bc4-kube-api-access-jcjq7\") pod \"cert-manager-webhook-6486ffd997-tssd9\" (UID: \"dac8a5cc-34f6-4f86-8a4c-9f8273878bc4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:07.086746 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:07.086692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dac8a5cc-34f6-4f86-8a4c-9f8273878bc4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tssd9\" (UID: \"dac8a5cc-34f6-4f86-8a4c-9f8273878bc4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:07.086836 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:07.086763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcjq7\" (UniqueName: \"kubernetes.io/projected/dac8a5cc-34f6-4f86-8a4c-9f8273878bc4-kube-api-access-jcjq7\") pod \"cert-manager-webhook-6486ffd997-tssd9\" (UID: \"dac8a5cc-34f6-4f86-8a4c-9f8273878bc4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:07.156474 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:07.156447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:07.422236 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:07.422206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tssd9"]
Sep 15 09:20:07.426190 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:20:07.426163    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddac8a5cc_34f6_4f86_8a4c_9f8273878bc4.slice/crio-32524caad676be6756b7438fe111e3e75ed1b73954c1e0f50a1b15cd7fab2e6a WatchSource:0}: Error finding container 32524caad676be6756b7438fe111e3e75ed1b73954c1e0f50a1b15cd7fab2e6a: Status 404 returned error can't find the container with id 32524caad676be6756b7438fe111e3e75ed1b73954c1e0f50a1b15cd7fab2e6a
Sep 15 09:20:07.428736 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:07.428717    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:20:08.249062 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:08.249022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9" event={"ID":"dac8a5cc-34f6-4f86-8a4c-9f8273878bc4","Type":"ContainerStarted","Data":"32524caad676be6756b7438fe111e3e75ed1b73954c1e0f50a1b15cd7fab2e6a"}
Sep 15 09:20:11.263293 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:11.263254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9" event={"ID":"dac8a5cc-34f6-4f86-8a4c-9f8273878bc4","Type":"ContainerStarted","Data":"99d353f226ec89503cd0e378bb5bdb0b943af3eb25b2247736631f7bc3cc2d77"}
Sep 15 09:20:11.263763 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:11.263361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:11.279243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:11.279200    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9" podStartSLOduration=2.171128642 podStartE2EDuration="5.279185668s" totalImagesPullingTime="3.108057026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:20:11.279132549 +0000 UTC m=+375.108924451" watchObservedRunningTime="2026-09-15 09:20:11.279185668 +0000 UTC m=+375.108977571"
Sep 15 09:20:17.268577 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:17.268550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-tssd9"
Sep 15 09:20:38.989724 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:38.989693    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"]
Sep 15 09:20:38.992848 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:38.992832    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:38.996318 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:38.996155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jtvl9\""
Sep 15 09:20:38.996318 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:38.996169    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 09:20:38.996318 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:38.996155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 09:20:38.996566 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:38.996165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 09:20:38.996566 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:38.996546    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 09:20:38.996678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:38.996606    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 09:20:39.003754 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.003733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"]
Sep 15 09:20:39.014239 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.014217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglk8\" (UniqueName: \"kubernetes.io/projected/35884fd4-cd31-4d87-880e-c17679682226-kube-api-access-pglk8\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.014346 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.014254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35884fd4-cd31-4d87-880e-c17679682226-metrics-cert\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.014410 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.014372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35884fd4-cd31-4d87-880e-c17679682226-cert\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.014410 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.014402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/35884fd4-cd31-4d87-880e-c17679682226-manager-config\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.115612 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.115583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglk8\" (UniqueName: \"kubernetes.io/projected/35884fd4-cd31-4d87-880e-c17679682226-kube-api-access-pglk8\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.115752 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.115621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35884fd4-cd31-4d87-880e-c17679682226-metrics-cert\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.115752 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.115682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35884fd4-cd31-4d87-880e-c17679682226-cert\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.115752 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.115710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/35884fd4-cd31-4d87-880e-c17679682226-manager-config\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.116376 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.116354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/35884fd4-cd31-4d87-880e-c17679682226-manager-config\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.118031 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.118009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35884fd4-cd31-4d87-880e-c17679682226-cert\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.118108 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.118077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35884fd4-cd31-4d87-880e-c17679682226-metrics-cert\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.126419 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.126399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglk8\" (UniqueName: \"kubernetes.io/projected/35884fd4-cd31-4d87-880e-c17679682226-kube-api-access-pglk8\") pod \"lws-controller-manager-6f69b6c8dc-nmrxg\" (UID: \"35884fd4-cd31-4d87-880e-c17679682226\") " pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.302144 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.302069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:39.585299 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:39.585198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"]
Sep 15 09:20:39.587876 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:20:39.587851    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35884fd4_cd31_4d87_880e_c17679682226.slice/crio-c1c7cdca6e9a3ec11a517196961a954175591409ddf317262c6f7cf1023207f7 WatchSource:0}: Error finding container c1c7cdca6e9a3ec11a517196961a954175591409ddf317262c6f7cf1023207f7: Status 404 returned error can't find the container with id c1c7cdca6e9a3ec11a517196961a954175591409ddf317262c6f7cf1023207f7
Sep 15 09:20:40.373153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:40.373105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg" event={"ID":"35884fd4-cd31-4d87-880e-c17679682226","Type":"ContainerStarted","Data":"c1c7cdca6e9a3ec11a517196961a954175591409ddf317262c6f7cf1023207f7"}
Sep 15 09:20:41.368953 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.368920    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"]
Sep 15 09:20:41.379768 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.379734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.382657 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.382629    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 09:20:41.383082 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.383052    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 09:20:41.383494 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.383468    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 09:20:41.383601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.383560    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d2wz4\""
Sep 15 09:20:41.383668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.383631    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 09:20:41.392072 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.392048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"]
Sep 15 09:20:41.430543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.430513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7lgp\" (UniqueName: \"kubernetes.io/projected/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-kube-api-access-m7lgp\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.430680 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.430557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-tmp\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.430680 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.430636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-manifests\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.430772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.430692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.430772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.430713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.531754 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.531719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.531934 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.531764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.531934 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.531817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7lgp\" (UniqueName: \"kubernetes.io/projected/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-kube-api-access-m7lgp\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.531934 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.531844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-tmp\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.532116 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.531968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-manifests\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.532404 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.532352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-manifests\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.534388 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.534361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-tmp\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.534388 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.534371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.534597 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.534577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.541283 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.541258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7lgp\" (UniqueName: \"kubernetes.io/projected/0d3fc6aa-0b4e-4401-a35a-876bf7c445f0-kube-api-access-m7lgp\") pod \"opendatahub-operator-controller-manager-767c4c8b76-mbwh7\" (UID: \"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0\") " pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:41.694299 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:41.694266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:42.155432 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:42.155265    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"]
Sep 15 09:20:42.158504 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:20:42.158478    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3fc6aa_0b4e_4401_a35a_876bf7c445f0.slice/crio-0c3fe8df48db8dbf170bd07b04535a9b21e44198ed5eb1321b0622f6b4d7dd80 WatchSource:0}: Error finding container 0c3fe8df48db8dbf170bd07b04535a9b21e44198ed5eb1321b0622f6b4d7dd80: Status 404 returned error can't find the container with id 0c3fe8df48db8dbf170bd07b04535a9b21e44198ed5eb1321b0622f6b4d7dd80
Sep 15 09:20:42.384578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:42.384493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7" event={"ID":"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0","Type":"ContainerStarted","Data":"0c3fe8df48db8dbf170bd07b04535a9b21e44198ed5eb1321b0622f6b4d7dd80"}
Sep 15 09:20:42.385999 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:42.385969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg" event={"ID":"35884fd4-cd31-4d87-880e-c17679682226","Type":"ContainerStarted","Data":"856436012c8ebb2d78d5171b3d960b4ce8e74ccca2d6824cff3eeb23a235c717"}
Sep 15 09:20:42.386123 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:42.386083    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:42.406973 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:42.406932    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg" podStartSLOduration=2.080999141 podStartE2EDuration="4.40691754s" totalImagesPullingTime="2.325918399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:20:42.40457435 +0000 UTC m=+406.234366263" watchObservedRunningTime="2026-09-15 09:20:42.40691754 +0000 UTC m=+406.236709443"
Sep 15 09:20:45.402354 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:45.402241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7" event={"ID":"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0","Type":"ContainerStarted","Data":"28f883bbe35fdd103fbd8100f6e678c89172372b76e9566f1206863143dfbead"}
Sep 15 09:20:46.407981 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:46.407944    2579 generic.go:356] "Generic (PLEG): container finished" podID="0d3fc6aa-0b4e-4401-a35a-876bf7c445f0" containerID="28f883bbe35fdd103fbd8100f6e678c89172372b76e9566f1206863143dfbead" exitCode=0
Sep 15 09:20:46.408383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:46.408036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7" event={"ID":"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0","Type":"ContainerDied","Data":"28f883bbe35fdd103fbd8100f6e678c89172372b76e9566f1206863143dfbead"}
Sep 15 09:20:47.416132 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:47.416082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7" event={"ID":"0d3fc6aa-0b4e-4401-a35a-876bf7c445f0","Type":"ContainerStarted","Data":"177f359d798d434ed87c7ccc117f73aca41a4edc1c4f492720c70a9ddbea5d5c"}
Sep 15 09:20:47.416634 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:47.416190    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:20:47.446231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:47.446185    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7" podStartSLOduration=2.105545881 podStartE2EDuration="6.446168517s" totalImagesPullingTime="3.337921379s" totalInitContainerRuntime="1.002701257s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:20:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:20:47.444625742 +0000 UTC m=+411.274417659" watchObservedRunningTime="2026-09-15 09:20:47.446168517 +0000 UTC m=+411.275960420"
Sep 15 09:20:53.394569 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:53.394537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f69b6c8dc-nmrxg"
Sep 15 09:20:58.422438 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:20:58.422401    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-767c4c8b76-mbwh7"
Sep 15 09:21:00.716058 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.716021    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"]
Sep 15 09:21:00.725347 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.725315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:00.728366 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.728312    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 09:21:00.728541 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.728525    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 09:21:00.728594 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.728578    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 09:21:00.729182 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.729169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-djxzm\""
Sep 15 09:21:00.729811 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.729797    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 09:21:00.740404 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.740382    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"]
Sep 15 09:21:00.766967 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.766938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f27408-50b5-4cc0-8de5-514f9cd191a2-tls-certs\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:00.767058 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.767018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11f27408-50b5-4cc0-8de5-514f9cd191a2-tmp\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:00.767150 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.767131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6c6\" (UniqueName: \"kubernetes.io/projected/11f27408-50b5-4cc0-8de5-514f9cd191a2-kube-api-access-2s6c6\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:00.868446 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.868416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11f27408-50b5-4cc0-8de5-514f9cd191a2-tmp\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:00.868553 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.868450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6c6\" (UniqueName: \"kubernetes.io/projected/11f27408-50b5-4cc0-8de5-514f9cd191a2-kube-api-access-2s6c6\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:00.868553 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.868509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f27408-50b5-4cc0-8de5-514f9cd191a2-tls-certs\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:00.868667 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:21:00.868597    2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 15 09:21:00.868667 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:21:00.868648    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f27408-50b5-4cc0-8de5-514f9cd191a2-tls-certs podName:11f27408-50b5-4cc0-8de5-514f9cd191a2 nodeName:}" failed. No retries permitted until 2026-09-15 09:21:01.368634099 +0000 UTC m=+425.198425979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/11f27408-50b5-4cc0-8de5-514f9cd191a2-tls-certs") pod "kube-auth-proxy-6df85cfc68-5tsp2" (UID: "11f27408-50b5-4cc0-8de5-514f9cd191a2") : secret "kube-auth-proxy-tls" not found
Sep 15 09:21:00.870538 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.870517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11f27408-50b5-4cc0-8de5-514f9cd191a2-tmp\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:00.876392 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:00.876370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6c6\" (UniqueName: \"kubernetes.io/projected/11f27408-50b5-4cc0-8de5-514f9cd191a2-kube-api-access-2s6c6\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:01.372652 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:01.372621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f27408-50b5-4cc0-8de5-514f9cd191a2-tls-certs\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:01.374912 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:01.374893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f27408-50b5-4cc0-8de5-514f9cd191a2-tls-certs\") pod \"kube-auth-proxy-6df85cfc68-5tsp2\" (UID: \"11f27408-50b5-4cc0-8de5-514f9cd191a2\") " pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:01.635787 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:01.635701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"
Sep 15 09:21:01.908807 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:01.908777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2"]
Sep 15 09:21:01.911754 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:21:01.911728    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f27408_50b5_4cc0_8de5_514f9cd191a2.slice/crio-99eb3060bf113e5acf702a925a44d3fdceb834f9c2de693cf707eb4f79773f10 WatchSource:0}: Error finding container 99eb3060bf113e5acf702a925a44d3fdceb834f9c2de693cf707eb4f79773f10: Status 404 returned error can't find the container with id 99eb3060bf113e5acf702a925a44d3fdceb834f9c2de693cf707eb4f79773f10
Sep 15 09:21:02.482842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:02.482805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2" event={"ID":"11f27408-50b5-4cc0-8de5-514f9cd191a2","Type":"ContainerStarted","Data":"99eb3060bf113e5acf702a925a44d3fdceb834f9c2de693cf707eb4f79773f10"}
Sep 15 09:21:05.504095 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:05.504047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2" event={"ID":"11f27408-50b5-4cc0-8de5-514f9cd191a2","Type":"ContainerStarted","Data":"df85613ea8a490d4288805f0153bf89fba63493b68058f89f424accf7c091aa1"}
Sep 15 09:21:05.521215 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:21:05.521134    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6df85cfc68-5tsp2" podStartSLOduration=2.535018938 podStartE2EDuration="5.521106244s" totalImagesPullingTime="2.986087306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:21:05.518433316 +0000 UTC m=+429.348225231" watchObservedRunningTime="2026-09-15 09:21:05.521106244 +0000 UTC m=+429.350898149"
Sep 15 09:22:49.632411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.632294    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"]
Sep 15 09:22:49.635760 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.635741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"
Sep 15 09:22:49.638255 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.638232    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 09:22:49.638830 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.638808    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 09:22:49.638937 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.638862    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 09:22:49.639140 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.639125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lwgsk\""
Sep 15 09:22:49.647104 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.647078    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"]
Sep 15 09:22:49.738678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.738648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4gl9\" (UniqueName: \"kubernetes.io/projected/a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2-kube-api-access-r4gl9\") pod \"dns-operator-controller-manager-7b84cd7c9c-5zn66\" (UID: \"a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"
Sep 15 09:22:49.775895 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.775860    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"]
Sep 15 09:22:49.779172 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.779153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:49.781679 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.781656    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 09:22:49.781679 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.781674    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 09:22:49.781855 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.781662    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wl9lf\""
Sep 15 09:22:49.790757 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.790739    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"]
Sep 15 09:22:49.839586 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.839564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gl9\" (UniqueName: \"kubernetes.io/projected/a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2-kube-api-access-r4gl9\") pod \"dns-operator-controller-manager-7b84cd7c9c-5zn66\" (UID: \"a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"
Sep 15 09:22:49.847785 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.847761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4gl9\" (UniqueName: \"kubernetes.io/projected/a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2-kube-api-access-r4gl9\") pod \"dns-operator-controller-manager-7b84cd7c9c-5zn66\" (UID: \"a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"
Sep 15 09:22:49.940688 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.940661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f275b\" (UniqueName: \"kubernetes.io/projected/98d6af91-4a3f-4737-bbfa-6d7832b93b6f-kube-api-access-f275b\") pod \"kuadrant-console-plugin-fbdd4f445-srwxw\" (UID: \"98d6af91-4a3f-4737-bbfa-6d7832b93b6f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:49.940803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.940709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/98d6af91-4a3f-4737-bbfa-6d7832b93b6f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-srwxw\" (UID: \"98d6af91-4a3f-4737-bbfa-6d7832b93b6f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:49.940803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.940740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d6af91-4a3f-4737-bbfa-6d7832b93b6f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-srwxw\" (UID: \"98d6af91-4a3f-4737-bbfa-6d7832b93b6f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:49.948132 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:49.948103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"
Sep 15 09:22:50.041731 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.041702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/98d6af91-4a3f-4737-bbfa-6d7832b93b6f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-srwxw\" (UID: \"98d6af91-4a3f-4737-bbfa-6d7832b93b6f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:50.041879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.041743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d6af91-4a3f-4737-bbfa-6d7832b93b6f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-srwxw\" (UID: \"98d6af91-4a3f-4737-bbfa-6d7832b93b6f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:50.041879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.041836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f275b\" (UniqueName: \"kubernetes.io/projected/98d6af91-4a3f-4737-bbfa-6d7832b93b6f-kube-api-access-f275b\") pod \"kuadrant-console-plugin-fbdd4f445-srwxw\" (UID: \"98d6af91-4a3f-4737-bbfa-6d7832b93b6f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:50.042451 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.042430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/98d6af91-4a3f-4737-bbfa-6d7832b93b6f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-srwxw\" (UID: \"98d6af91-4a3f-4737-bbfa-6d7832b93b6f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:50.043975 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.043954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d6af91-4a3f-4737-bbfa-6d7832b93b6f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-srwxw\" (UID: \"98d6af91-4a3f-4737-bbfa-6d7832b93b6f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:50.049946 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.049920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f275b\" (UniqueName: \"kubernetes.io/projected/98d6af91-4a3f-4737-bbfa-6d7832b93b6f-kube-api-access-f275b\") pod \"kuadrant-console-plugin-fbdd4f445-srwxw\" (UID: \"98d6af91-4a3f-4737-bbfa-6d7832b93b6f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:50.088325 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.088305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"
Sep 15 09:22:50.222716 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.222694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"]
Sep 15 09:22:50.225106 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:22:50.225078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b9e2e1_fffb_4c6b_96da_2bb9ffda68b2.slice/crio-c67b39be3fda32e259152cecf0f1b4e39132d8ace6862b595528e4c1ddda93b7 WatchSource:0}: Error finding container c67b39be3fda32e259152cecf0f1b4e39132d8ace6862b595528e4c1ddda93b7: Status 404 returned error can't find the container with id c67b39be3fda32e259152cecf0f1b4e39132d8ace6862b595528e4c1ddda93b7
Sep 15 09:22:50.358950 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.358921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw"]
Sep 15 09:22:50.361799 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:22:50.361776    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98d6af91_4a3f_4737_bbfa_6d7832b93b6f.slice/crio-ab77c7850b1446e688744d4bd5f5f151ff886e75c5b0f929e11e84e26e3a7ad0 WatchSource:0}: Error finding container ab77c7850b1446e688744d4bd5f5f151ff886e75c5b0f929e11e84e26e3a7ad0: Status 404 returned error can't find the container with id ab77c7850b1446e688744d4bd5f5f151ff886e75c5b0f929e11e84e26e3a7ad0
Sep 15 09:22:50.958394 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.958358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw" event={"ID":"98d6af91-4a3f-4737-bbfa-6d7832b93b6f","Type":"ContainerStarted","Data":"ab77c7850b1446e688744d4bd5f5f151ff886e75c5b0f929e11e84e26e3a7ad0"}
Sep 15 09:22:50.959672 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:50.959648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66" event={"ID":"a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2","Type":"ContainerStarted","Data":"c67b39be3fda32e259152cecf0f1b4e39132d8ace6862b595528e4c1ddda93b7"}
Sep 15 09:22:55.989461 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:55.989424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw" event={"ID":"98d6af91-4a3f-4737-bbfa-6d7832b93b6f","Type":"ContainerStarted","Data":"987981ae91de762629356c55c3a34237e75df739d4975d5fbfec84ee5df1f5c2"}
Sep 15 09:22:55.991170 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:55.991142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66" event={"ID":"a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2","Type":"ContainerStarted","Data":"74cd7193e88d60ab1b75236c4fcbd55ed99bbef3bf5df4b0e55b5eb57f0f6261"}
Sep 15 09:22:55.991305 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:55.991288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"
Sep 15 09:22:56.005145 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:56.005104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-srwxw" podStartSLOduration=1.705274448 podStartE2EDuration="7.005080022s" totalImagesPullingTime="5.299805574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:22:56.004131696 +0000 UTC m=+539.833923599" watchObservedRunningTime="2026-09-15 09:22:56.005080022 +0000 UTC m=+539.834871924"
Sep 15 09:22:56.021069 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:22:56.021026    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66" podStartSLOduration=1.6520873310000002 podStartE2EDuration="7.021008016s" totalImagesPullingTime="5.368920685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:22:56.020228925 +0000 UTC m=+539.850020829" watchObservedRunningTime="2026-09-15 09:22:56.021008016 +0000 UTC m=+539.850799920"
Sep 15 09:23:00.588025 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.587987    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"]
Sep 15 09:23:00.592357 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.592312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:00.595062 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.595026    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2rwrt\""
Sep 15 09:23:00.609033 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.608998    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"]
Sep 15 09:23:00.634238 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.634143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxrp4\" (UniqueName: \"kubernetes.io/projected/6f8a4b66-330b-4182-92d8-e773fab1ac4a-kube-api-access-lxrp4\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6xjn2\" (UID: \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:00.634497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.634355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f8a4b66-330b-4182-92d8-e773fab1ac4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6xjn2\" (UID: \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:00.735548 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.735499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f8a4b66-330b-4182-92d8-e773fab1ac4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6xjn2\" (UID: \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:00.735758 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.735572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxrp4\" (UniqueName: \"kubernetes.io/projected/6f8a4b66-330b-4182-92d8-e773fab1ac4a-kube-api-access-lxrp4\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6xjn2\" (UID: \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:00.735841 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.735818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f8a4b66-330b-4182-92d8-e773fab1ac4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6xjn2\" (UID: \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:00.746420 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.746382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxrp4\" (UniqueName: \"kubernetes.io/projected/6f8a4b66-330b-4182-92d8-e773fab1ac4a-kube-api-access-lxrp4\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6xjn2\" (UID: \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:00.903535 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:00.903440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:01.214415 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:01.214382    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"]
Sep 15 09:23:01.217816 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:23:01.217786    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8a4b66_330b_4182_92d8_e773fab1ac4a.slice/crio-7a65bfd83281e1bedd8ca5f54554950977bf6545476dd8c4efce59f3c1f20098 WatchSource:0}: Error finding container 7a65bfd83281e1bedd8ca5f54554950977bf6545476dd8c4efce59f3c1f20098: Status 404 returned error can't find the container with id 7a65bfd83281e1bedd8ca5f54554950977bf6545476dd8c4efce59f3c1f20098
Sep 15 09:23:02.018785 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:02.018744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2" event={"ID":"6f8a4b66-330b-4182-92d8-e773fab1ac4a","Type":"ContainerStarted","Data":"7a65bfd83281e1bedd8ca5f54554950977bf6545476dd8c4efce59f3c1f20098"}
Sep 15 09:23:05.038622 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:05.038531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2" event={"ID":"6f8a4b66-330b-4182-92d8-e773fab1ac4a","Type":"ContainerStarted","Data":"7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876"}
Sep 15 09:23:05.039016 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:05.038705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:05.059355 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:05.059292    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2" podStartSLOduration=1.584221017 podStartE2EDuration="5.059274654s" totalImagesPullingTime="3.475053637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:23:05.056908823 +0000 UTC m=+548.886700728" watchObservedRunningTime="2026-09-15 09:23:05.059274654 +0000 UTC m=+548.889066557"
Sep 15 09:23:06.997065 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:06.997037    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5zn66"
Sep 15 09:23:16.045281 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:16.045242    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:28.558658 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:28.558617    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"]
Sep 15 09:23:28.559199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:28.558844    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2" podUID="6f8a4b66-330b-4182-92d8-e773fab1ac4a" containerName="manager" containerID="cri-o://7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876" gracePeriod=10
Sep 15 09:23:28.794092 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:28.794070    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:28.870977 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:28.870906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f8a4b66-330b-4182-92d8-e773fab1ac4a-extensions-socket-volume\") pod \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\" (UID: \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\") "
Sep 15 09:23:28.870977 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:28.870940    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxrp4\" (UniqueName: \"kubernetes.io/projected/6f8a4b66-330b-4182-92d8-e773fab1ac4a-kube-api-access-lxrp4\") pod \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\" (UID: \"6f8a4b66-330b-4182-92d8-e773fab1ac4a\") "
Sep 15 09:23:28.871423 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:28.871397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8a4b66-330b-4182-92d8-e773fab1ac4a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6f8a4b66-330b-4182-92d8-e773fab1ac4a" (UID: "6f8a4b66-330b-4182-92d8-e773fab1ac4a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:23:28.872878 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:28.872847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8a4b66-330b-4182-92d8-e773fab1ac4a-kube-api-access-lxrp4" (OuterVolumeSpecName: "kube-api-access-lxrp4") pod "6f8a4b66-330b-4182-92d8-e773fab1ac4a" (UID: "6f8a4b66-330b-4182-92d8-e773fab1ac4a"). InnerVolumeSpecName "kube-api-access-lxrp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:23:28.971641 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:28.971617    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxrp4\" (UniqueName: \"kubernetes.io/projected/6f8a4b66-330b-4182-92d8-e773fab1ac4a-kube-api-access-lxrp4\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:23:28.971641 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:28.971640    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f8a4b66-330b-4182-92d8-e773fab1ac4a-extensions-socket-volume\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:23:29.146440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:29.146365    2579 generic.go:356] "Generic (PLEG): container finished" podID="6f8a4b66-330b-4182-92d8-e773fab1ac4a" containerID="7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876" exitCode=0
Sep 15 09:23:29.146440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:29.146401    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"
Sep 15 09:23:29.146440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:29.146424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2" event={"ID":"6f8a4b66-330b-4182-92d8-e773fab1ac4a","Type":"ContainerDied","Data":"7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876"}
Sep 15 09:23:29.146618 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:29.146473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2" event={"ID":"6f8a4b66-330b-4182-92d8-e773fab1ac4a","Type":"ContainerDied","Data":"7a65bfd83281e1bedd8ca5f54554950977bf6545476dd8c4efce59f3c1f20098"}
Sep 15 09:23:29.146618 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:29.146495    2579 scope.go:117] "RemoveContainer" containerID="7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876"
Sep 15 09:23:29.155039 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:29.155021    2579 scope.go:117] "RemoveContainer" containerID="7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876"
Sep 15 09:23:29.155321 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:23:29.155302    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876\": container with ID starting with 7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876 not found: ID does not exist" containerID="7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876"
Sep 15 09:23:29.155406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:29.155329    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876"} err="failed to get container status \"7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876\": rpc error: code = NotFound desc = could not find container \"7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876\": container with ID starting with 7bb737b09ee441d260a7556c0eebcea203ddc2124e650f1a8968acfaef765876 not found: ID does not exist"
Sep 15 09:23:29.166479 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:29.166455    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"]
Sep 15 09:23:29.172947 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:29.172928    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6xjn2"]
Sep 15 09:23:30.674385 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:30.674330    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f8a4b66-330b-4182-92d8-e773fab1ac4a" path="/var/lib/kubelet/pods/6f8a4b66-330b-4182-92d8-e773fab1ac4a/volumes"
Sep 15 09:23:46.286680 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.286640    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:23:46.287269 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.287188    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8a4b66-330b-4182-92d8-e773fab1ac4a" containerName="manager"
Sep 15 09:23:46.287269 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.287209    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8a4b66-330b-4182-92d8-e773fab1ac4a" containerName="manager"
Sep 15 09:23:46.287423 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.287284    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8a4b66-330b-4182-92d8-e773fab1ac4a" containerName="manager"
Sep 15 09:23:46.291817 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.291772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:23:46.293915 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.293896    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 09:23:46.297423 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.297400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:23:46.321814 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.321788    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:23:46.415967 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.415939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzbh\" (UniqueName: \"kubernetes.io/projected/1ac1449c-8d70-47ac-bd7c-a802cb83deb3-kube-api-access-2hzbh\") pod \"limitador-limitador-84994f7ffc-8czjn\" (UID: \"1ac1449c-8d70-47ac-bd7c-a802cb83deb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:23:46.416083 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.415982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1ac1449c-8d70-47ac-bd7c-a802cb83deb3-config-file\") pod \"limitador-limitador-84994f7ffc-8czjn\" (UID: \"1ac1449c-8d70-47ac-bd7c-a802cb83deb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:23:46.516572 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.516539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzbh\" (UniqueName: \"kubernetes.io/projected/1ac1449c-8d70-47ac-bd7c-a802cb83deb3-kube-api-access-2hzbh\") pod \"limitador-limitador-84994f7ffc-8czjn\" (UID: \"1ac1449c-8d70-47ac-bd7c-a802cb83deb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:23:46.516690 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.516587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1ac1449c-8d70-47ac-bd7c-a802cb83deb3-config-file\") pod \"limitador-limitador-84994f7ffc-8czjn\" (UID: \"1ac1449c-8d70-47ac-bd7c-a802cb83deb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:23:46.517159 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.517139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1ac1449c-8d70-47ac-bd7c-a802cb83deb3-config-file\") pod \"limitador-limitador-84994f7ffc-8czjn\" (UID: \"1ac1449c-8d70-47ac-bd7c-a802cb83deb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:23:46.524076 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.524046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzbh\" (UniqueName: \"kubernetes.io/projected/1ac1449c-8d70-47ac-bd7c-a802cb83deb3-kube-api-access-2hzbh\") pod \"limitador-limitador-84994f7ffc-8czjn\" (UID: \"1ac1449c-8d70-47ac-bd7c-a802cb83deb3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:23:46.603177 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.603125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:23:46.686989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.686958    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v9xzk"]
Sep 15 09:23:46.690536 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.690516    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk"
Sep 15 09:23:46.692549 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.692523    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qf6s4\""
Sep 15 09:23:46.696789 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.696767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v9xzk"]
Sep 15 09:23:46.819589 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.819557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44jpn\" (UniqueName: \"kubernetes.io/projected/ffeed5ac-bf14-4168-9af8-ca6dbda78032-kube-api-access-44jpn\") pod \"authorino-7bcfd8f9cd-v9xzk\" (UID: \"ffeed5ac-bf14-4168-9af8-ca6dbda78032\") " pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk"
Sep 15 09:23:46.868826 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.868806    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:23:46.870697 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:23:46.870659    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac1449c_8d70_47ac_bd7c_a802cb83deb3.slice/crio-e76594581182ee51fb375820bd23414c382b0def336cf07222c00bf3e37ca86f WatchSource:0}: Error finding container e76594581182ee51fb375820bd23414c382b0def336cf07222c00bf3e37ca86f: Status 404 returned error can't find the container with id e76594581182ee51fb375820bd23414c382b0def336cf07222c00bf3e37ca86f
Sep 15 09:23:46.920369 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.920324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44jpn\" (UniqueName: \"kubernetes.io/projected/ffeed5ac-bf14-4168-9af8-ca6dbda78032-kube-api-access-44jpn\") pod \"authorino-7bcfd8f9cd-v9xzk\" (UID: \"ffeed5ac-bf14-4168-9af8-ca6dbda78032\") " pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk"
Sep 15 09:23:46.927059 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:46.927036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44jpn\" (UniqueName: \"kubernetes.io/projected/ffeed5ac-bf14-4168-9af8-ca6dbda78032-kube-api-access-44jpn\") pod \"authorino-7bcfd8f9cd-v9xzk\" (UID: \"ffeed5ac-bf14-4168-9af8-ca6dbda78032\") " pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk"
Sep 15 09:23:47.000378 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:47.000354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk"
Sep 15 09:23:47.227436 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:47.227398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn" event={"ID":"1ac1449c-8d70-47ac-bd7c-a802cb83deb3","Type":"ContainerStarted","Data":"e76594581182ee51fb375820bd23414c382b0def336cf07222c00bf3e37ca86f"}
Sep 15 09:23:47.269098 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:47.269075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v9xzk"]
Sep 15 09:23:47.271397 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:23:47.271375    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffeed5ac_bf14_4168_9af8_ca6dbda78032.slice/crio-d8f3b96e5a21c955a9c2fe94a0b11f38af4d6f47b67c320be0523d7832aeb5cd WatchSource:0}: Error finding container d8f3b96e5a21c955a9c2fe94a0b11f38af4d6f47b67c320be0523d7832aeb5cd: Status 404 returned error can't find the container with id d8f3b96e5a21c955a9c2fe94a0b11f38af4d6f47b67c320be0523d7832aeb5cd
Sep 15 09:23:48.234822 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:48.234783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn" event={"ID":"1ac1449c-8d70-47ac-bd7c-a802cb83deb3","Type":"ContainerStarted","Data":"a89f0176451d345985fc877957bc5d4b77188da8822f1b5f6040e57108365a08"}
Sep 15 09:23:48.235265 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:48.234887    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:23:48.236007 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:48.235978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk" event={"ID":"ffeed5ac-bf14-4168-9af8-ca6dbda78032","Type":"ContainerStarted","Data":"d8f3b96e5a21c955a9c2fe94a0b11f38af4d6f47b67c320be0523d7832aeb5cd"}
Sep 15 09:23:48.251442 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:48.251392    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn" podStartSLOduration=1.428987574 podStartE2EDuration="2.251378464s" totalImagesPullingTime="822.39089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:23:48.248852201 +0000 UTC m=+592.078644105" watchObservedRunningTime="2026-09-15 09:23:48.251378464 +0000 UTC m=+592.081170371"
Sep 15 09:23:49.241860 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:49.241781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk" event={"ID":"ffeed5ac-bf14-4168-9af8-ca6dbda78032","Type":"ContainerStarted","Data":"6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f"}
Sep 15 09:23:49.254971 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:49.254927    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk" podStartSLOduration=1.602692009 podStartE2EDuration="3.25491196s" totalImagesPullingTime="1.652219951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:23:49.254366161 +0000 UTC m=+593.084158060" watchObservedRunningTime="2026-09-15 09:23:49.25491196 +0000 UTC m=+593.084703863"
Sep 15 09:23:49.710386 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:49.710351    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v9xzk"]
Sep 15 09:23:51.251406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:51.251364    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk" podUID="ffeed5ac-bf14-4168-9af8-ca6dbda78032" containerName="authorino" containerID="cri-o://6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f" gracePeriod=30
Sep 15 09:23:51.492642 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:51.492615    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk"
Sep 15 09:23:51.658657 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:51.658632    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44jpn\" (UniqueName: \"kubernetes.io/projected/ffeed5ac-bf14-4168-9af8-ca6dbda78032-kube-api-access-44jpn\") pod \"ffeed5ac-bf14-4168-9af8-ca6dbda78032\" (UID: \"ffeed5ac-bf14-4168-9af8-ca6dbda78032\") "
Sep 15 09:23:51.660634 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:51.660610    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffeed5ac-bf14-4168-9af8-ca6dbda78032-kube-api-access-44jpn" (OuterVolumeSpecName: "kube-api-access-44jpn") pod "ffeed5ac-bf14-4168-9af8-ca6dbda78032" (UID: "ffeed5ac-bf14-4168-9af8-ca6dbda78032"). InnerVolumeSpecName "kube-api-access-44jpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:23:51.759690 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:51.759662    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44jpn\" (UniqueName: \"kubernetes.io/projected/ffeed5ac-bf14-4168-9af8-ca6dbda78032-kube-api-access-44jpn\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:23:52.256398 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.256368    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffeed5ac-bf14-4168-9af8-ca6dbda78032" containerID="6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f" exitCode=0
Sep 15 09:23:52.256761 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.256409    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk"
Sep 15 09:23:52.256761 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.256452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk" event={"ID":"ffeed5ac-bf14-4168-9af8-ca6dbda78032","Type":"ContainerDied","Data":"6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f"}
Sep 15 09:23:52.256761 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.256490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-v9xzk" event={"ID":"ffeed5ac-bf14-4168-9af8-ca6dbda78032","Type":"ContainerDied","Data":"d8f3b96e5a21c955a9c2fe94a0b11f38af4d6f47b67c320be0523d7832aeb5cd"}
Sep 15 09:23:52.256761 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.256508    2579 scope.go:117] "RemoveContainer" containerID="6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f"
Sep 15 09:23:52.265091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.265060    2579 scope.go:117] "RemoveContainer" containerID="6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f"
Sep 15 09:23:52.270135 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:23:52.270097    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f\": container with ID starting with 6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f not found: ID does not exist" containerID="6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f"
Sep 15 09:23:52.270284 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.270259    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f"} err="failed to get container status \"6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f\": rpc error: code = NotFound desc = could not find container \"6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f\": container with ID starting with 6df6977ce6e86a2b19cc2ab19f5806dc2762c63a2db66367e3cb4c20abce977f not found: ID does not exist"
Sep 15 09:23:52.276471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.276452    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v9xzk"]
Sep 15 09:23:52.279767 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.279749    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v9xzk"]
Sep 15 09:23:52.674257 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:52.674217    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffeed5ac-bf14-4168-9af8-ca6dbda78032" path="/var/lib/kubelet/pods/ffeed5ac-bf14-4168-9af8-ca6dbda78032/volumes"
Sep 15 09:23:59.242779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:23:59.242749    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8czjn"
Sep 15 09:24:14.394266 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.394232    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-2hdp2"]
Sep 15 09:24:14.394722 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.394575    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffeed5ac-bf14-4168-9af8-ca6dbda78032" containerName="authorino"
Sep 15 09:24:14.394722 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.394587    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffeed5ac-bf14-4168-9af8-ca6dbda78032" containerName="authorino"
Sep 15 09:24:14.394722 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.394656    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffeed5ac-bf14-4168-9af8-ca6dbda78032" containerName="authorino"
Sep 15 09:24:14.399100 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.399082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2hdp2"
Sep 15 09:24:14.401694 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.401667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qf6s4\""
Sep 15 09:24:14.402910 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.402887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-2hdp2"]
Sep 15 09:24:14.437579 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.437557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglzc\" (UniqueName: \"kubernetes.io/projected/fe8025c2-9370-4e90-b96e-c8e5f104d22c-kube-api-access-mglzc\") pod \"authorino-557f876f69-2hdp2\" (UID: \"fe8025c2-9370-4e90-b96e-c8e5f104d22c\") " pod="kuadrant-system/authorino-557f876f69-2hdp2"
Sep 15 09:24:14.520471 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.520445    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2hdp2"]
Sep 15 09:24:14.520658 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:24:14.520640    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mglzc], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-2hdp2" podUID="fe8025c2-9370-4e90-b96e-c8e5f104d22c"
Sep 15 09:24:14.538012 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.537987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglzc\" (UniqueName: \"kubernetes.io/projected/fe8025c2-9370-4e90-b96e-c8e5f104d22c-kube-api-access-mglzc\") pod \"authorino-557f876f69-2hdp2\" (UID: \"fe8025c2-9370-4e90-b96e-c8e5f104d22c\") " pod="kuadrant-system/authorino-557f876f69-2hdp2"
Sep 15 09:24:14.546867 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.546848    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d6sfs"]
Sep 15 09:24:14.549174 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.549150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglzc\" (UniqueName: \"kubernetes.io/projected/fe8025c2-9370-4e90-b96e-c8e5f104d22c-kube-api-access-mglzc\") pod \"authorino-557f876f69-2hdp2\" (UID: \"fe8025c2-9370-4e90-b96e-c8e5f104d22c\") " pod="kuadrant-system/authorino-557f876f69-2hdp2"
Sep 15 09:24:14.550116 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.550101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:14.552013 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.551997    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 09:24:14.556362 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.556330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d6sfs"]
Sep 15 09:24:14.618472 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.618444    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d6sfs"]
Sep 15 09:24:14.618671 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:24:14.618650    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jnbqx tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-d6sfs" podUID="88765f8f-24b6-40d2-a67d-9667b627a13b"
Sep 15 09:24:14.638784 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.638757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88765f8f-24b6-40d2-a67d-9667b627a13b-tls-cert\") pod \"authorino-6d4dfc649f-d6sfs\" (UID: \"88765f8f-24b6-40d2-a67d-9667b627a13b\") " pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:14.638875 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.638803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnbqx\" (UniqueName: \"kubernetes.io/projected/88765f8f-24b6-40d2-a67d-9667b627a13b-kube-api-access-jnbqx\") pod \"authorino-6d4dfc649f-d6sfs\" (UID: \"88765f8f-24b6-40d2-a67d-9667b627a13b\") " pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:14.739750 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.739725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88765f8f-24b6-40d2-a67d-9667b627a13b-tls-cert\") pod \"authorino-6d4dfc649f-d6sfs\" (UID: \"88765f8f-24b6-40d2-a67d-9667b627a13b\") " pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:14.739856 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.739778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbqx\" (UniqueName: \"kubernetes.io/projected/88765f8f-24b6-40d2-a67d-9667b627a13b-kube-api-access-jnbqx\") pod \"authorino-6d4dfc649f-d6sfs\" (UID: \"88765f8f-24b6-40d2-a67d-9667b627a13b\") " pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:14.742206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.742181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88765f8f-24b6-40d2-a67d-9667b627a13b-tls-cert\") pod \"authorino-6d4dfc649f-d6sfs\" (UID: \"88765f8f-24b6-40d2-a67d-9667b627a13b\") " pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:14.746880 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:14.746856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnbqx\" (UniqueName: \"kubernetes.io/projected/88765f8f-24b6-40d2-a67d-9667b627a13b-kube-api-access-jnbqx\") pod \"authorino-6d4dfc649f-d6sfs\" (UID: \"88765f8f-24b6-40d2-a67d-9667b627a13b\") " pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:15.357283 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.357253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:15.357462 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.357256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2hdp2"
Sep 15 09:24:15.364271 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.364246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:15.367116 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.367101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2hdp2"
Sep 15 09:24:15.444166 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.444140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnbqx\" (UniqueName: \"kubernetes.io/projected/88765f8f-24b6-40d2-a67d-9667b627a13b-kube-api-access-jnbqx\") pod \"88765f8f-24b6-40d2-a67d-9667b627a13b\" (UID: \"88765f8f-24b6-40d2-a67d-9667b627a13b\") "
Sep 15 09:24:15.444533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.444203    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mglzc\" (UniqueName: \"kubernetes.io/projected/fe8025c2-9370-4e90-b96e-c8e5f104d22c-kube-api-access-mglzc\") pod \"fe8025c2-9370-4e90-b96e-c8e5f104d22c\" (UID: \"fe8025c2-9370-4e90-b96e-c8e5f104d22c\") "
Sep 15 09:24:15.444533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.444220    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88765f8f-24b6-40d2-a67d-9667b627a13b-tls-cert\") pod \"88765f8f-24b6-40d2-a67d-9667b627a13b\" (UID: \"88765f8f-24b6-40d2-a67d-9667b627a13b\") "
Sep 15 09:24:15.446138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.446116    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88765f8f-24b6-40d2-a67d-9667b627a13b-kube-api-access-jnbqx" (OuterVolumeSpecName: "kube-api-access-jnbqx") pod "88765f8f-24b6-40d2-a67d-9667b627a13b" (UID: "88765f8f-24b6-40d2-a67d-9667b627a13b"). InnerVolumeSpecName "kube-api-access-jnbqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:24:15.446273 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.446251    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8025c2-9370-4e90-b96e-c8e5f104d22c-kube-api-access-mglzc" (OuterVolumeSpecName: "kube-api-access-mglzc") pod "fe8025c2-9370-4e90-b96e-c8e5f104d22c" (UID: "fe8025c2-9370-4e90-b96e-c8e5f104d22c"). InnerVolumeSpecName "kube-api-access-mglzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:24:15.446375 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.446355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88765f8f-24b6-40d2-a67d-9667b627a13b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "88765f8f-24b6-40d2-a67d-9667b627a13b" (UID: "88765f8f-24b6-40d2-a67d-9667b627a13b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:24:15.545481 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.545455    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnbqx\" (UniqueName: \"kubernetes.io/projected/88765f8f-24b6-40d2-a67d-9667b627a13b-kube-api-access-jnbqx\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:24:15.545481 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.545476    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mglzc\" (UniqueName: \"kubernetes.io/projected/fe8025c2-9370-4e90-b96e-c8e5f104d22c-kube-api-access-mglzc\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:24:15.545481 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:15.545485    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88765f8f-24b6-40d2-a67d-9667b627a13b-tls-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:24:16.361505 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:16.361473    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d6sfs"
Sep 15 09:24:16.361701 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:16.361475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2hdp2"
Sep 15 09:24:16.397178 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:16.397144    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d6sfs"]
Sep 15 09:24:16.402611 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:16.402573    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d6sfs"]
Sep 15 09:24:16.431452 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:16.430405    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2hdp2"]
Sep 15 09:24:16.435243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:16.435220    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-2hdp2"]
Sep 15 09:24:16.675805 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:16.675776    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88765f8f-24b6-40d2-a67d-9667b627a13b" path="/var/lib/kubelet/pods/88765f8f-24b6-40d2-a67d-9667b627a13b/volumes"
Sep 15 09:24:16.676165 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:16.676011    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8025c2-9370-4e90-b96e-c8e5f104d22c" path="/var/lib/kubelet/pods/fe8025c2-9370-4e90-b96e-c8e5f104d22c/volumes"
Sep 15 09:24:20.761578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.761501    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-94dcc9b77-vt79h"]
Sep 15 09:24:20.768155 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.768133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:20.770637 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.770609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 09:24:20.770749 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.770659    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 09:24:20.770805 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.770611    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 09:24:20.771263 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.771237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lqk66\""
Sep 15 09:24:20.778545 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.778518    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94dcc9b77-vt79h"]
Sep 15 09:24:20.888545 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.888519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cch9w\" (UniqueName: \"kubernetes.io/projected/223c9ef9-a6d9-4944-9b76-f9d3abc70787-kube-api-access-cch9w\") pod \"maas-controller-94dcc9b77-vt79h\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") " pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:20.888656 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.888552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-webhook-cert\") pod \"maas-controller-94dcc9b77-vt79h\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") " pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:20.888656 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.888630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-metrics-certs\") pod \"maas-controller-94dcc9b77-vt79h\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") " pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:20.989891 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.989860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cch9w\" (UniqueName: \"kubernetes.io/projected/223c9ef9-a6d9-4944-9b76-f9d3abc70787-kube-api-access-cch9w\") pod \"maas-controller-94dcc9b77-vt79h\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") " pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:20.990024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.989897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-webhook-cert\") pod \"maas-controller-94dcc9b77-vt79h\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") " pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:20.990024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.989922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-metrics-certs\") pod \"maas-controller-94dcc9b77-vt79h\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") " pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:20.992271 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.992244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-webhook-cert\") pod \"maas-controller-94dcc9b77-vt79h\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") " pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:20.992438 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.992416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-metrics-certs\") pod \"maas-controller-94dcc9b77-vt79h\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") " pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:20.997287 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:20.997265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cch9w\" (UniqueName: \"kubernetes.io/projected/223c9ef9-a6d9-4944-9b76-f9d3abc70787-kube-api-access-cch9w\") pod \"maas-controller-94dcc9b77-vt79h\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") " pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:21.084622 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:21.084569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:21.354010 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:21.353984    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-94dcc9b77-vt79h"]
Sep 15 09:24:21.356547 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:24:21.356514    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223c9ef9_a6d9_4944_9b76_f9d3abc70787.slice/crio-dc841cb380dd96e7c0809060aa9945974810576a11ca2223ddc5c0a55b1577f7 WatchSource:0}: Error finding container dc841cb380dd96e7c0809060aa9945974810576a11ca2223ddc5c0a55b1577f7: Status 404 returned error can't find the container with id dc841cb380dd96e7c0809060aa9945974810576a11ca2223ddc5c0a55b1577f7
Sep 15 09:24:21.383257 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:21.383226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcc9b77-vt79h" event={"ID":"223c9ef9-a6d9-4944-9b76-f9d3abc70787","Type":"ContainerStarted","Data":"dc841cb380dd96e7c0809060aa9945974810576a11ca2223ddc5c0a55b1577f7"}
Sep 15 09:24:24.400610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:24.400546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcc9b77-vt79h" event={"ID":"223c9ef9-a6d9-4944-9b76-f9d3abc70787","Type":"ContainerStarted","Data":"a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300"}
Sep 15 09:24:24.401134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:24.400652    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:24.417214 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:24.417162    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-94dcc9b77-vt79h" podStartSLOduration=2.145897251 podStartE2EDuration="4.4171423s" totalImagesPullingTime="2.271245049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:24:24.41616161 +0000 UTC m=+628.245953517" watchObservedRunningTime="2026-09-15 09:24:24.4171423 +0000 UTC m=+628.246934203"
Sep 15 09:24:34.911000 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:34.910969    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-94dcc9b77-vt79h"]
Sep 15 09:24:34.911451 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:34.911170    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-94dcc9b77-vt79h" podUID="223c9ef9-a6d9-4944-9b76-f9d3abc70787" containerName="manager" containerID="cri-o://a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300" gracePeriod=30
Sep 15 09:24:34.913269 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:34.913245    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:35.143068 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.143045    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:35.309399 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.309312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-webhook-cert\") pod \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") "
Sep 15 09:24:35.309399 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.309391    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-metrics-certs\") pod \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") "
Sep 15 09:24:35.309604 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.309411    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cch9w\" (UniqueName: \"kubernetes.io/projected/223c9ef9-a6d9-4944-9b76-f9d3abc70787-kube-api-access-cch9w\") pod \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\" (UID: \"223c9ef9-a6d9-4944-9b76-f9d3abc70787\") "
Sep 15 09:24:35.311383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.311348    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "223c9ef9-a6d9-4944-9b76-f9d3abc70787" (UID: "223c9ef9-a6d9-4944-9b76-f9d3abc70787"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:24:35.311558 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.311536    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "223c9ef9-a6d9-4944-9b76-f9d3abc70787" (UID: "223c9ef9-a6d9-4944-9b76-f9d3abc70787"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:24:35.311627 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.311563    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223c9ef9-a6d9-4944-9b76-f9d3abc70787-kube-api-access-cch9w" (OuterVolumeSpecName: "kube-api-access-cch9w") pod "223c9ef9-a6d9-4944-9b76-f9d3abc70787" (UID: "223c9ef9-a6d9-4944-9b76-f9d3abc70787"). InnerVolumeSpecName "kube-api-access-cch9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:24:35.410185 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.410156    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-webhook-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:24:35.410185 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.410179    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223c9ef9-a6d9-4944-9b76-f9d3abc70787-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:24:35.410304 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.410191    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cch9w\" (UniqueName: \"kubernetes.io/projected/223c9ef9-a6d9-4944-9b76-f9d3abc70787-kube-api-access-cch9w\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:24:35.451696 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.451671    2579 generic.go:356] "Generic (PLEG): container finished" podID="223c9ef9-a6d9-4944-9b76-f9d3abc70787" containerID="a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300" exitCode=0
Sep 15 09:24:35.451775 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.451714    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-94dcc9b77-vt79h"
Sep 15 09:24:35.451775 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.451720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcc9b77-vt79h" event={"ID":"223c9ef9-a6d9-4944-9b76-f9d3abc70787","Type":"ContainerDied","Data":"a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300"}
Sep 15 09:24:35.451775 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.451760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-94dcc9b77-vt79h" event={"ID":"223c9ef9-a6d9-4944-9b76-f9d3abc70787","Type":"ContainerDied","Data":"dc841cb380dd96e7c0809060aa9945974810576a11ca2223ddc5c0a55b1577f7"}
Sep 15 09:24:35.451884 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.451781    2579 scope.go:117] "RemoveContainer" containerID="a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300"
Sep 15 09:24:35.460499 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.460481    2579 scope.go:117] "RemoveContainer" containerID="a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300"
Sep 15 09:24:35.460757 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:24:35.460740    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300\": container with ID starting with a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300 not found: ID does not exist" containerID="a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300"
Sep 15 09:24:35.460810 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.460762    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300"} err="failed to get container status \"a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300\": rpc error: code = NotFound desc = could not find container \"a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300\": container with ID starting with a00d2947f43838c7ce244a8de284f049255306d70074ddd229c2c1ed4f227300 not found: ID does not exist"
Sep 15 09:24:35.473511 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.473492    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-94dcc9b77-vt79h"]
Sep 15 09:24:35.477047 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:35.477026    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-94dcc9b77-vt79h"]
Sep 15 09:24:36.674863 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:36.674823    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223c9ef9-a6d9-4944-9b76-f9d3abc70787" path="/var/lib/kubelet/pods/223c9ef9-a6d9-4944-9b76-f9d3abc70787/volumes"
Sep 15 09:24:55.771388 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.771353    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"]
Sep 15 09:24:55.773698 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.771696    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223c9ef9-a6d9-4944-9b76-f9d3abc70787" containerName="manager"
Sep 15 09:24:55.773698 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.771707    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="223c9ef9-a6d9-4944-9b76-f9d3abc70787" containerName="manager"
Sep 15 09:24:55.773698 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.771755    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="223c9ef9-a6d9-4944-9b76-f9d3abc70787" containerName="manager"
Sep 15 09:24:55.774635 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.774620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.776795 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.776766    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 09:24:55.776795 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.776779    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hb4ml\""
Sep 15 09:24:55.776795 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.776779    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 09:24:55.777594 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.777574    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 09:24:55.777698 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.777618    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 09:24:55.785160 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.785138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"]
Sep 15 09:24:55.871729 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.871701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n48k\" (UniqueName: \"kubernetes.io/projected/1f0d04e9-a9a4-4913-9350-00ce30b70788-kube-api-access-5n48k\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.871871 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.871760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-metrics-certs\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.871871 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.871805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d04e9-a9a4-4913-9350-00ce30b70788-openshift-service-ca\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.871871 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.871850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-maas-api-tls\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.972253 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.972223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n48k\" (UniqueName: \"kubernetes.io/projected/1f0d04e9-a9a4-4913-9350-00ce30b70788-kube-api-access-5n48k\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.972442 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.972281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-metrics-certs\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.972442 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.972299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d04e9-a9a4-4913-9350-00ce30b70788-openshift-service-ca\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.972442 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.972349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-maas-api-tls\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.972442 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:24:55.972414    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 15 09:24:55.972606 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:24:55.972500    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-metrics-certs podName:1f0d04e9-a9a4-4913-9350-00ce30b70788 nodeName:}" failed. No retries permitted until 2026-09-15 09:24:56.472478305 +0000 UTC m=+660.302270201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-metrics-certs") pod "maas-api-68fc5fb999-7zz48" (UID: "1f0d04e9-a9a4-4913-9350-00ce30b70788") : secret "maas-api-metrics-tls" not found
Sep 15 09:24:55.973020 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.973000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d04e9-a9a4-4913-9350-00ce30b70788-openshift-service-ca\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.975286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.975260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-maas-api-tls\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:55.984137 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:55.984115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n48k\" (UniqueName: \"kubernetes.io/projected/1f0d04e9-a9a4-4913-9350-00ce30b70788-kube-api-access-5n48k\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:56.291734 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:56.291703    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:24:56.476179 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:56.476150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-metrics-certs\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:56.478440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:56.478410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-metrics-certs\") pod \"maas-api-68fc5fb999-7zz48\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") " pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:56.686413 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:56.686389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hb4ml\""
Sep 15 09:24:56.695134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:56.695117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:56.967148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:56.967122    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"]
Sep 15 09:24:56.969124 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:24:56.969097    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0d04e9_a9a4_4913_9350_00ce30b70788.slice/crio-a8b3aea485d7c2e63eaf7c01d2993a340d004e2b64429650c2e2461865a4fb71 WatchSource:0}: Error finding container a8b3aea485d7c2e63eaf7c01d2993a340d004e2b64429650c2e2461865a4fb71: Status 404 returned error can't find the container with id a8b3aea485d7c2e63eaf7c01d2993a340d004e2b64429650c2e2461865a4fb71
Sep 15 09:24:57.551794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:57.551754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48" event={"ID":"1f0d04e9-a9a4-4913-9350-00ce30b70788","Type":"ContainerStarted","Data":"a8b3aea485d7c2e63eaf7c01d2993a340d004e2b64429650c2e2461865a4fb71"}
Sep 15 09:24:59.562704 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:59.562667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48" event={"ID":"1f0d04e9-a9a4-4913-9350-00ce30b70788","Type":"ContainerStarted","Data":"9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f"}
Sep 15 09:24:59.563100 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:59.562787    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:24:59.578803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:24:59.578755    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48" podStartSLOduration=2.809751856 podStartE2EDuration="4.578741248s" totalImagesPullingTime="1.768989392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:24:59.576553894 +0000 UTC m=+663.406345797" watchObservedRunningTime="2026-09-15 09:24:59.578741248 +0000 UTC m=+663.408533144"
Sep 15 09:25:05.572553 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:05.572524    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:25:22.906829 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.906795    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"]
Sep 15 09:25:22.909991 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.909972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:22.912212 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.912190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 09:25:22.912317 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.912191    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 09:25:22.912989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.912968    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 09:25:22.913083 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.913006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d8dng\""
Sep 15 09:25:22.918240 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.918216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"]
Sep 15 09:25:22.983053 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.983024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:22.983206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.983072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjp7\" (UniqueName: \"kubernetes.io/projected/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-kube-api-access-gcjp7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:22.983206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.983130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:22.983206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.983170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:22.983373 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.983221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:22.983373 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:22.983255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.083985 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.083955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.084145 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.084013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.084145 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.084049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.084145 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.084103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.084145 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.084138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjp7\" (UniqueName: \"kubernetes.io/projected/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-kube-api-access-gcjp7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.084362 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.084179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.084496 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.084465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.084582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.084501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.084582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.084474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.086243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.086219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.086565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.086547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.095525 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.095498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjp7\" (UniqueName: \"kubernetes.io/projected/aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03-kube-api-access-gcjp7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv\" (UID: \"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.220899 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.220834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:23.490190 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.490164    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"]
Sep 15 09:25:23.492244 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:25:23.492209    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeecbd62_6c59_4f8f_bc6e_0e01c65bcd03.slice/crio-edaa878e2808745a2ae9181144d29cd18bd13f80aed823a914392f33cc2303c9 WatchSource:0}: Error finding container edaa878e2808745a2ae9181144d29cd18bd13f80aed823a914392f33cc2303c9: Status 404 returned error can't find the container with id edaa878e2808745a2ae9181144d29cd18bd13f80aed823a914392f33cc2303c9
Sep 15 09:25:23.493634 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.493617    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:25:23.592836 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.592800    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:25:23.680569 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:23.680536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv" event={"ID":"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03","Type":"ContainerStarted","Data":"edaa878e2808745a2ae9181144d29cd18bd13f80aed823a914392f33cc2303c9"}
Sep 15 09:25:24.686460 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:24.686430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv" event={"ID":"aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03","Type":"ContainerStarted","Data":"d30db4abc46472a2c2323bb4523c0f7ccd05eabf3177d56f4ee0c9b718b3b8a6"}
Sep 15 09:25:24.702494 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:24.702445    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv" podStartSLOduration=1.611234286 podStartE2EDuration="2.702430992s" totalImagesPullingTime="1.091196706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:25:24.700574575 +0000 UTC m=+688.530366477" watchObservedRunningTime="2026-09-15 09:25:24.702430992 +0000 UTC m=+688.532222894"
Sep 15 09:25:30.705801 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:30.705770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:25:33.221817 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:33.221780    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:33.222162 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:33.221878    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:33.234045 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:33.234019    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:33.744112 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:33.744078    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv"
Sep 15 09:25:43.593987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:43.593953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:25:48.490315 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:48.490281    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:25:54.801461 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:25:54.801427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:26:08.292520 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.292478    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"]
Sep 15 09:26:08.298790 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.298753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.301438 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.301408    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 09:26:08.305880 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.305772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"]
Sep 15 09:26:08.364408 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.364357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.364408 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.364414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fws\" (UniqueName: \"kubernetes.io/projected/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-kube-api-access-87fws\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.364670 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.364453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.364670 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.364494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.364670 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.364522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.364670 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.364549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.465780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.465719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.465780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.465775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fws\" (UniqueName: \"kubernetes.io/projected/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-kube-api-access-87fws\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.466024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.465805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.466024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.465831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.466024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.465849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.466024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.465872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.466224 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.466199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.466224 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.466218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.466360 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.466282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.468302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.468264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.468552 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.468532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.474517 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.474485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fws\" (UniqueName: \"kubernetes.io/projected/0cc002ec-ba8f-42a4-ae54-43337c4ac31a-kube-api-access-87fws\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xckhd\" (UID: \"0cc002ec-ba8f-42a4-ae54-43337c4ac31a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.612438 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.612312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:08.887676 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:08.887644    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"]
Sep 15 09:26:08.890630 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:26:08.890591    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc002ec_ba8f_42a4_ae54_43337c4ac31a.slice/crio-ebfad3a646a6a5a412931195b1ca5f6946d42ba63c304845d7ce22eb0aa3ea06 WatchSource:0}: Error finding container ebfad3a646a6a5a412931195b1ca5f6946d42ba63c304845d7ce22eb0aa3ea06: Status 404 returned error can't find the container with id ebfad3a646a6a5a412931195b1ca5f6946d42ba63c304845d7ce22eb0aa3ea06
Sep 15 09:26:09.900967 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:09.900927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd" event={"ID":"0cc002ec-ba8f-42a4-ae54-43337c4ac31a","Type":"ContainerStarted","Data":"7429e3cf5a81a0fa356a5d03ec76864d9b4003118abba14464eb2b33dddf3cd1"}
Sep 15 09:26:09.901370 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:09.900984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd" event={"ID":"0cc002ec-ba8f-42a4-ae54-43337c4ac31a","Type":"ContainerStarted","Data":"ebfad3a646a6a5a412931195b1ca5f6946d42ba63c304845d7ce22eb0aa3ea06"}
Sep 15 09:26:09.918409 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:09.918352    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd" podStartSLOduration=1.691701139 podStartE2EDuration="1.91832211s" totalImagesPullingTime="226.620971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:26:09.916452132 +0000 UTC m=+733.746244036" watchObservedRunningTime="2026-09-15 09:26:09.91832211 +0000 UTC m=+733.748114013"
Sep 15 09:26:10.099611 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:10.099575    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:26:17.096251 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.096209    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"]
Sep 15 09:26:17.100118 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.100087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.102648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.102620    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 09:26:17.110261 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.110225    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"]
Sep 15 09:26:17.146157 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.146113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjrd\" (UniqueName: \"kubernetes.io/projected/28c3ebc8-4517-4d4b-938b-dba6596d36bb-kube-api-access-jzjrd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.146383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.146172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.146383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.146224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.146383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.146240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.146383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.146304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.146532 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.146419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c3ebc8-4517-4d4b-938b-dba6596d36bb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.247644 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.247598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c3ebc8-4517-4d4b-938b-dba6596d36bb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.247862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.247659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjrd\" (UniqueName: \"kubernetes.io/projected/28c3ebc8-4517-4d4b-938b-dba6596d36bb-kube-api-access-jzjrd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.247862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.247707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.247862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.247741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.247862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.247763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.247862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.247804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.248239 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.248210    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.248301 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.248239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.248370 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.248302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.250132 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.250099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c3ebc8-4517-4d4b-938b-dba6596d36bb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.250308 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.250292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c3ebc8-4517-4d4b-938b-dba6596d36bb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.255186 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.255154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjrd\" (UniqueName: \"kubernetes.io/projected/28c3ebc8-4517-4d4b-938b-dba6596d36bb-kube-api-access-jzjrd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd\" (UID: \"28c3ebc8-4517-4d4b-938b-dba6596d36bb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.412040 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.412000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:17.704893 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.704666    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"]
Sep 15 09:26:17.714146 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:26:17.714114    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28c3ebc8_4517_4d4b_938b_dba6596d36bb.slice/crio-f8c08911bd1c2caa8eeefc675ed6fee21da5502f6e93162979029f2557be0d42 WatchSource:0}: Error finding container f8c08911bd1c2caa8eeefc675ed6fee21da5502f6e93162979029f2557be0d42: Status 404 returned error can't find the container with id f8c08911bd1c2caa8eeefc675ed6fee21da5502f6e93162979029f2557be0d42
Sep 15 09:26:17.940748 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:17.940717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd" event={"ID":"28c3ebc8-4517-4d4b-938b-dba6596d36bb","Type":"ContainerStarted","Data":"f8c08911bd1c2caa8eeefc675ed6fee21da5502f6e93162979029f2557be0d42"}
Sep 15 09:26:18.613267 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:18.613234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:18.613651 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:18.613316    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:18.625492 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:18.625470    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:18.946782 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:18.946748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd" event={"ID":"28c3ebc8-4517-4d4b-938b-dba6596d36bb","Type":"ContainerStarted","Data":"ee9d13a227328468a10de234a954afcba21edcb9efc512c227e30142b7f7f4c4"}
Sep 15 09:26:18.958629 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:18.958605    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xckhd"
Sep 15 09:26:18.967904 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:18.967839    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd" podStartSLOduration=1.788315291 podStartE2EDuration="1.967823736s" totalImagesPullingTime="179.508445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:26:18.965548444 +0000 UTC m=+742.795340350" watchObservedRunningTime="2026-09-15 09:26:18.967823736 +0000 UTC m=+742.797615638"
Sep 15 09:26:19.399177 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:19.399087    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:26:27.412130 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:27.412096    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:27.412130 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:27.412135    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:27.424564 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:27.424537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:26:28.000490 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:26:28.000461    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd"
Sep 15 09:27:53.132549 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:53.132509    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-173ccf"]
Sep 15 09:27:53.136020 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:53.136002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-173ccf"
Sep 15 09:27:53.138095 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:53.138077    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-88s6f\""
Sep 15 09:27:53.142074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:53.142052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-173ccf"]
Sep 15 09:27:53.272668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:53.272625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsfj\" (UniqueName: \"kubernetes.io/projected/2e3cb204-6222-41fc-aeef-0656e0b97008-kube-api-access-hzsfj\") pod \"test-curl-7567-173ccf\" (UID: \"2e3cb204-6222-41fc-aeef-0656e0b97008\") " pod="opendatahub/test-curl-7567-173ccf"
Sep 15 09:27:53.373554 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:53.373508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsfj\" (UniqueName: \"kubernetes.io/projected/2e3cb204-6222-41fc-aeef-0656e0b97008-kube-api-access-hzsfj\") pod \"test-curl-7567-173ccf\" (UID: \"2e3cb204-6222-41fc-aeef-0656e0b97008\") " pod="opendatahub/test-curl-7567-173ccf"
Sep 15 09:27:53.380886 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:53.380862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsfj\" (UniqueName: \"kubernetes.io/projected/2e3cb204-6222-41fc-aeef-0656e0b97008-kube-api-access-hzsfj\") pod \"test-curl-7567-173ccf\" (UID: \"2e3cb204-6222-41fc-aeef-0656e0b97008\") " pod="opendatahub/test-curl-7567-173ccf"
Sep 15 09:27:53.445710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:53.445676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-173ccf"
Sep 15 09:27:53.713578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:53.713548    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-173ccf"]
Sep 15 09:27:53.716400 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:27:53.716369    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e3cb204_6222_41fc_aeef_0656e0b97008.slice/crio-44b5bd3ecb824bc45eac3869175385e1a364f17c822eef77550c361b4b066aa2 WatchSource:0}: Error finding container 44b5bd3ecb824bc45eac3869175385e1a364f17c822eef77550c361b4b066aa2: Status 404 returned error can't find the container with id 44b5bd3ecb824bc45eac3869175385e1a364f17c822eef77550c361b4b066aa2
Sep 15 09:27:54.391700 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:54.391657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-173ccf" event={"ID":"2e3cb204-6222-41fc-aeef-0656e0b97008","Type":"ContainerStarted","Data":"44b5bd3ecb824bc45eac3869175385e1a364f17c822eef77550c361b4b066aa2"}
Sep 15 09:27:54.542724 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:54.542684    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:27:55.520582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.520547    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"]
Sep 15 09:27:55.533180 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.532867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.533180 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.533094    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"]
Sep 15 09:27:55.535226 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.535029    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7a31d6-serving-cert\""
Sep 15 09:27:55.595253 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.595221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.595434 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.595323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7dw\" (UniqueName: \"kubernetes.io/projected/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-kube-api-access-bn7dw\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.595434 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.595392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.595434 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.595429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-metrics-certs\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.695977 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.695951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.696077 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.695992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-metrics-certs\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.696077 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.696013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.696158 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.696105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7dw\" (UniqueName: \"kubernetes.io/projected/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-kube-api-access-bn7dw\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.696202 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:27:55.696169    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-serving-cert: secret "maas-api-e2e-inf-a-7a31d6-serving-cert" not found
Sep 15 09:27:55.696238 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:27:55.696224    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-maas-api-tls podName:2cc1c1f0-bac5-4ad4-b618-34afa2df2afc nodeName:}" failed. No retries permitted until 2026-09-15 09:27:56.196208349 +0000 UTC m=+840.026000230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-maas-api-tls") pod "maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx" (UID: "2cc1c1f0-bac5-4ad4-b618-34afa2df2afc") : secret "maas-api-e2e-inf-a-7a31d6-serving-cert" not found
Sep 15 09:27:55.696578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.696554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.698189 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.698168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-metrics-certs\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:55.704557 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:55.704539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7dw\" (UniqueName: \"kubernetes.io/projected/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-kube-api-access-bn7dw\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:56.200566 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.200531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:56.202943 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.202915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:56.403513 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.403477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-173ccf" event={"ID":"2e3cb204-6222-41fc-aeef-0656e0b97008","Type":"ContainerStarted","Data":"9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516"}
Sep 15 09:27:56.418209 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.418163    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7567-173ccf" podStartSLOduration=1.522265556 podStartE2EDuration="3.418149271s" totalImagesPullingTime="1.895883715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:27:56.417162711 +0000 UTC m=+840.246954614" watchObservedRunningTime="2026-09-15 09:27:56.418149271 +0000 UTC m=+840.247941174"
Sep 15 09:27:56.445224 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.445196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:56.492055 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.491984    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"]
Sep 15 09:27:56.497756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.497268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.500861 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.500680    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c183a5-serving-cert\""
Sep 15 09:27:56.510590 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.506905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"]
Sep 15 09:27:56.606277 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.606240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.606277 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.606278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-metrics-certs\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.606695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.606374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59dbc\" (UniqueName: \"kubernetes.io/projected/52397807-e1d5-4274-95f7-bf88ddd0d345-kube-api-access-59dbc\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.606695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.606477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52397807-e1d5-4274-95f7-bf88ddd0d345-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.707086 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.707002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52397807-e1d5-4274-95f7-bf88ddd0d345-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.707086 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.707038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.707270 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.707086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-metrics-certs\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.707270 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.707153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59dbc\" (UniqueName: \"kubernetes.io/projected/52397807-e1d5-4274-95f7-bf88ddd0d345-kube-api-access-59dbc\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.707789 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.707767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52397807-e1d5-4274-95f7-bf88ddd0d345-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.709630 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.709611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-metrics-certs\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.711176 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.711156    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c183a5-serving-cert\""
Sep 15 09:27:56.717294 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.717264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59dbc\" (UniqueName: \"kubernetes.io/projected/52397807-e1d5-4274-95f7-bf88ddd0d345-kube-api-access-59dbc\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.719422 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.719400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:56.724189 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.724165    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"]
Sep 15 09:27:56.726413 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:27:56.726387    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1c1f0_bac5_4ad4_b618_34afa2df2afc.slice/crio-7031f0819546c92b7ef04bfe056667a24444cd669131ba44005110405020f054 WatchSource:0}: Error finding container 7031f0819546c92b7ef04bfe056667a24444cd669131ba44005110405020f054: Status 404 returned error can't find the container with id 7031f0819546c92b7ef04bfe056667a24444cd669131ba44005110405020f054
Sep 15 09:27:56.816866 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:56.816827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:57.014663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.012653    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-173ccf"]
Sep 15 09:27:57.021058 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.020900    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-173ccf"]
Sep 15 09:27:57.102364 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.102319    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"]
Sep 15 09:27:57.104450 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:27:57.104426    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52397807_e1d5_4274_95f7_bf88ddd0d345.slice/crio-7a20957fab462a4bc56388f703a5cc3798ff37de99669f92d2d660ad15eb226e WatchSource:0}: Error finding container 7a20957fab462a4bc56388f703a5cc3798ff37de99669f92d2d660ad15eb226e: Status 404 returned error can't find the container with id 7a20957fab462a4bc56388f703a5cc3798ff37de99669f92d2d660ad15eb226e
Sep 15 09:27:57.139529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.139492    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-80489d"]
Sep 15 09:27:57.139929 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.139916    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3cb204-6222-41fc-aeef-0656e0b97008" containerName="test-curl-7567-173ccf"
Sep 15 09:27:57.139972 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.139931    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3cb204-6222-41fc-aeef-0656e0b97008" containerName="test-curl-7567-173ccf"
Sep 15 09:27:57.140008 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.139990    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e3cb204-6222-41fc-aeef-0656e0b97008" containerName="test-curl-7567-173ccf"
Sep 15 09:27:57.144620 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.144572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-80489d"
Sep 15 09:27:57.147467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.147348    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-80489d"]
Sep 15 09:27:57.212427 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.212396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmf8\" (UniqueName: \"kubernetes.io/projected/8c8eead1-d4f5-473d-929e-66318bd32d1c-kube-api-access-pxmf8\") pod \"test-curl-7567-80489d\" (UID: \"8c8eead1-d4f5-473d-929e-66318bd32d1c\") " pod="opendatahub/test-curl-7567-80489d"
Sep 15 09:27:57.313997 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.313894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmf8\" (UniqueName: \"kubernetes.io/projected/8c8eead1-d4f5-473d-929e-66318bd32d1c-kube-api-access-pxmf8\") pod \"test-curl-7567-80489d\" (UID: \"8c8eead1-d4f5-473d-929e-66318bd32d1c\") " pod="opendatahub/test-curl-7567-80489d"
Sep 15 09:27:57.322166 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.322134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmf8\" (UniqueName: \"kubernetes.io/projected/8c8eead1-d4f5-473d-929e-66318bd32d1c-kube-api-access-pxmf8\") pod \"test-curl-7567-80489d\" (UID: \"8c8eead1-d4f5-473d-929e-66318bd32d1c\") " pod="opendatahub/test-curl-7567-80489d"
Sep 15 09:27:57.411121 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.411083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8" event={"ID":"52397807-e1d5-4274-95f7-bf88ddd0d345","Type":"ContainerStarted","Data":"7a20957fab462a4bc56388f703a5cc3798ff37de99669f92d2d660ad15eb226e"}
Sep 15 09:27:57.412809 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.412774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx" event={"ID":"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc","Type":"ContainerStarted","Data":"b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76"}
Sep 15 09:27:57.412961 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.412835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx" event={"ID":"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc","Type":"ContainerStarted","Data":"7031f0819546c92b7ef04bfe056667a24444cd669131ba44005110405020f054"}
Sep 15 09:27:57.413033 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:27:57.412996    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7567-173ccf" container="test-curl-7567-173ccf"
Sep 15 09:27:57.431428 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.431356    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx" podStartSLOduration=1.967265731 podStartE2EDuration="2.431322822s" totalImagesPullingTime="464.057091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:27:57.429619399 +0000 UTC m=+841.259411303" watchObservedRunningTime="2026-09-15 09:27:57.431322822 +0000 UTC m=+841.261114725"
Sep 15 09:27:57.456439 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.456401    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-80489d"
Sep 15 09:27:57.748037 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:57.748009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-80489d"]
Sep 15 09:27:57.750083 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:27:57.750054    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c8eead1_d4f5_473d_929e_66318bd32d1c.slice/crio-612cb52fa1d46f3cbd5c6b9dd752e36713cb8a8ce82180c7fa308bd7b5c45415 WatchSource:0}: Error finding container 612cb52fa1d46f3cbd5c6b9dd752e36713cb8a8ce82180c7fa308bd7b5c45415: Status 404 returned error can't find the container with id 612cb52fa1d46f3cbd5c6b9dd752e36713cb8a8ce82180c7fa308bd7b5c45415
Sep 15 09:27:58.418939 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:58.418898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8" event={"ID":"52397807-e1d5-4274-95f7-bf88ddd0d345","Type":"ContainerStarted","Data":"bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25"}
Sep 15 09:27:58.419149 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:58.419039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:27:58.420611 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:58.420584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-80489d" event={"ID":"8c8eead1-d4f5-473d-929e-66318bd32d1c","Type":"ContainerStarted","Data":"aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b"}
Sep 15 09:27:58.420755 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:58.420623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-80489d" event={"ID":"8c8eead1-d4f5-473d-929e-66318bd32d1c","Type":"ContainerStarted","Data":"612cb52fa1d46f3cbd5c6b9dd752e36713cb8a8ce82180c7fa308bd7b5c45415"}
Sep 15 09:27:58.420755 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:58.420699    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7567-173ccf" podUID="2e3cb204-6222-41fc-aeef-0656e0b97008" containerName="test-curl-7567-173ccf" containerID="cri-o://9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516" gracePeriod=2
Sep 15 09:27:58.420969 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:58.420928    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:27:58.434801 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:58.434727    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8" podStartSLOduration=2.029462453 podStartE2EDuration="2.434704457s" totalImagesPullingTime="405.242004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:27:58.433714337 +0000 UTC m=+842.263506240" watchObservedRunningTime="2026-09-15 09:27:58.434704457 +0000 UTC m=+842.264496361"
Sep 15 09:27:58.435589 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:27:58.435553    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-173ccf\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="2e3cb204-6222-41fc-aeef-0656e0b97008" pod="opendatahub/test-curl-7567-173ccf"
Sep 15 09:27:58.451161 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:58.451100    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7567-80489d" podStartSLOduration=1.451081679 podStartE2EDuration="1.451081679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:27:58.450503662 +0000 UTC m=+842.280295565" watchObservedRunningTime="2026-09-15 09:27:58.451081679 +0000 UTC m=+842.280873581"
Sep 15 09:27:59.028153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:59.028114    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-80489d"]
Sep 15 09:27:59.033979 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:27:59.033944    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-80489d"]
Sep 15 09:27:59.425850 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:27:59.425819    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7567-80489d" container="test-curl-7567-80489d"
Sep 15 09:28:00.431193 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:00.431143    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7567-80489d" podUID="8c8eead1-d4f5-473d-929e-66318bd32d1c" containerName="test-curl-7567-80489d" containerID="cri-o://aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b" gracePeriod=2
Sep 15 09:28:00.433053 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:28:00.433014    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-80489d\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="8c8eead1-d4f5-473d-929e-66318bd32d1c" pod="opendatahub/test-curl-7567-80489d"
Sep 15 09:28:00.597329 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:00.597296    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-173ccf"
Sep 15 09:28:00.599122 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:28:00.599097    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-80489d\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="8c8eead1-d4f5-473d-929e-66318bd32d1c" pod="opendatahub/test-curl-7567-80489d"
Sep 15 09:28:00.749928 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:00.749863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzsfj\" (UniqueName: \"kubernetes.io/projected/2e3cb204-6222-41fc-aeef-0656e0b97008-kube-api-access-hzsfj\") pod \"2e3cb204-6222-41fc-aeef-0656e0b97008\" (UID: \"2e3cb204-6222-41fc-aeef-0656e0b97008\") "
Sep 15 09:28:00.751835 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:00.751816    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3cb204-6222-41fc-aeef-0656e0b97008-kube-api-access-hzsfj" (OuterVolumeSpecName: "kube-api-access-hzsfj") pod "2e3cb204-6222-41fc-aeef-0656e0b97008" (UID: "2e3cb204-6222-41fc-aeef-0656e0b97008"). InnerVolumeSpecName "kube-api-access-hzsfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:28:00.850992 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:00.850960    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzsfj\" (UniqueName: \"kubernetes.io/projected/2e3cb204-6222-41fc-aeef-0656e0b97008-kube-api-access-hzsfj\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:28:01.296118 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.296088    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"]
Sep 15 09:28:01.296578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.296539    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8eead1-d4f5-473d-929e-66318bd32d1c" containerName="test-curl-7567-80489d"
Sep 15 09:28:01.296578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.296562    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8eead1-d4f5-473d-929e-66318bd32d1c" containerName="test-curl-7567-80489d"
Sep 15 09:28:01.296724 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.296699    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8eead1-d4f5-473d-929e-66318bd32d1c" containerName="test-curl-7567-80489d"
Sep 15 09:28:01.299817 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.299796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.302002 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.301975    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-e76bac-serving-cert\""
Sep 15 09:28:01.311298 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.311268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"]
Sep 15 09:28:01.436956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.436919    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e3cb204-6222-41fc-aeef-0656e0b97008" containerID="9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516" exitCode=137
Sep 15 09:28:01.436956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.436957    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-173ccf"
Sep 15 09:28:01.437443 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.437014    2579 scope.go:117] "RemoveContainer" containerID="9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516"
Sep 15 09:28:01.446584 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.446566    2579 scope.go:117] "RemoveContainer" containerID="9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516"
Sep 15 09:28:01.446910 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:28:01.446891    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516\": container with ID starting with 9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516 not found: ID does not exist" containerID="9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516"
Sep 15 09:28:01.446960 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.446921    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516"} err="failed to get container status \"9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516\": rpc error: code = NotFound desc = could not find container \"9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516\": container with ID starting with 9fa5a110fc67ec9465eeb2e9afe62e0adba6373c4a2eb7630b93a8d02a0a8516 not found: ID does not exist"
Sep 15 09:28:01.454887 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.454862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06008619-a502-4318-9a7c-78d6b2f2aafb-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.454989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.454906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jh9\" (UniqueName: \"kubernetes.io/projected/06008619-a502-4318-9a7c-78d6b2f2aafb-kube-api-access-s9jh9\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.454989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.454927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-metrics-certs\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.455091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.455001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.555925 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.555814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-metrics-certs\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.555925 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.555862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.556174 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.555951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06008619-a502-4318-9a7c-78d6b2f2aafb-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.556174 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.555987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jh9\" (UniqueName: \"kubernetes.io/projected/06008619-a502-4318-9a7c-78d6b2f2aafb-kube-api-access-s9jh9\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.556683 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.556653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06008619-a502-4318-9a7c-78d6b2f2aafb-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.558484 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.558459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.558597 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.558520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-metrics-certs\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.564483 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.564455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jh9\" (UniqueName: \"kubernetes.io/projected/06008619-a502-4318-9a7c-78d6b2f2aafb-kube-api-access-s9jh9\") pod \"maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.613592 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.613545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:01.890222 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:01.890124    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"]
Sep 15 09:28:01.893795 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:28:01.893759    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06008619_a502_4318_9a7c_78d6b2f2aafb.slice/crio-bc6ad7884027b870800ed87f21a9a433187f1800f9ecfe418b3d3d65ea0d2e49 WatchSource:0}: Error finding container bc6ad7884027b870800ed87f21a9a433187f1800f9ecfe418b3d3d65ea0d2e49: Status 404 returned error can't find the container with id bc6ad7884027b870800ed87f21a9a433187f1800f9ecfe418b3d3d65ea0d2e49
Sep 15 09:28:02.444777 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:02.444738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd" event={"ID":"06008619-a502-4318-9a7c-78d6b2f2aafb","Type":"ContainerStarted","Data":"f5b618b5d2af211f04a78d343da02022730dc1611f327e437a65dbfcff6beca8"}
Sep 15 09:28:02.445077 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:02.444834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd" event={"ID":"06008619-a502-4318-9a7c-78d6b2f2aafb","Type":"ContainerStarted","Data":"bc6ad7884027b870800ed87f21a9a433187f1800f9ecfe418b3d3d65ea0d2e49"}
Sep 15 09:28:02.445077 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:02.444864    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:02.461766 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:02.461722    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd" podStartSLOduration=1.132639128 podStartE2EDuration="1.461709091s" totalImagesPullingTime="329.069963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:28:02.46048001 +0000 UTC m=+846.290271914" watchObservedRunningTime="2026-09-15 09:28:02.461709091 +0000 UTC m=+846.291500993"
Sep 15 09:28:02.620819 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:02.620794    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-80489d"
Sep 15 09:28:02.676097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:02.676063    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e3cb204-6222-41fc-aeef-0656e0b97008" path="/var/lib/kubelet/pods/2e3cb204-6222-41fc-aeef-0656e0b97008/volumes"
Sep 15 09:28:02.767396 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:02.767293    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxmf8\" (UniqueName: \"kubernetes.io/projected/8c8eead1-d4f5-473d-929e-66318bd32d1c-kube-api-access-pxmf8\") pod \"8c8eead1-d4f5-473d-929e-66318bd32d1c\" (UID: \"8c8eead1-d4f5-473d-929e-66318bd32d1c\") "
Sep 15 09:28:02.769377 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:02.769352    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8eead1-d4f5-473d-929e-66318bd32d1c-kube-api-access-pxmf8" (OuterVolumeSpecName: "kube-api-access-pxmf8") pod "8c8eead1-d4f5-473d-929e-66318bd32d1c" (UID: "8c8eead1-d4f5-473d-929e-66318bd32d1c"). InnerVolumeSpecName "kube-api-access-pxmf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:28:02.868278 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:02.868249    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxmf8\" (UniqueName: \"kubernetes.io/projected/8c8eead1-d4f5-473d-929e-66318bd32d1c-kube-api-access-pxmf8\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:28:03.454583 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:03.454550    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c8eead1-d4f5-473d-929e-66318bd32d1c" containerID="aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b" exitCode=137
Sep 15 09:28:03.455084 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:03.454598    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-80489d"
Sep 15 09:28:03.455084 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:03.454597    2579 scope.go:117] "RemoveContainer" containerID="aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b"
Sep 15 09:28:03.463981 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:03.463964    2579 scope.go:117] "RemoveContainer" containerID="aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b"
Sep 15 09:28:03.464225 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:28:03.464207    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b\": container with ID starting with aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b not found: ID does not exist" containerID="aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b"
Sep 15 09:28:03.464275 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:03.464232    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b"} err="failed to get container status \"aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b\": rpc error: code = NotFound desc = could not find container \"aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b\": container with ID starting with aa0b995ca186208efe61fb7b1204513f2a524327a950d6f6ad44512d0b32850b not found: ID does not exist"
Sep 15 09:28:04.430956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:04.430925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:28:04.431360 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:04.431328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:28:04.675285 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:04.675252    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c8eead1-d4f5-473d-929e-66318bd32d1c" path="/var/lib/kubelet/pods/8c8eead1-d4f5-473d-929e-66318bd32d1c/volumes"
Sep 15 09:28:06.604470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.604438    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"]
Sep 15 09:28:06.609504 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.609480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.612137 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.611945    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-3b0480-serving-cert\""
Sep 15 09:28:06.618457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.618433    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"]
Sep 15 09:28:06.706017 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.705987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-maas-api-tls\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.706017 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.706018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-metrics-certs\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.706216 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.706045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-openshift-service-ca\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.706216 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.706071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ltd\" (UniqueName: \"kubernetes.io/projected/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-kube-api-access-f9ltd\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.807021 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.806985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-maas-api-tls\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.807021 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.807016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-metrics-certs\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.807237 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.807134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-openshift-service-ca\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.807237 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.807174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ltd\" (UniqueName: \"kubernetes.io/projected/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-kube-api-access-f9ltd\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.807836 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.807804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-openshift-service-ca\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.809514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.809491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-metrics-certs\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.809602 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.809497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-maas-api-tls\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.814840 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.814820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ltd\" (UniqueName: \"kubernetes.io/projected/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-kube-api-access-f9ltd\") pod \"maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:06.922659 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:06.922624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:07.194401 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.194370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"]
Sep 15 09:28:07.195972 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:28:07.195942    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8548bb_41f7_4e15_81ef_019fc4e1b93b.slice/crio-a48bc15d0eaff812eeb69777395ed09dc1fc05ca551e364418939d4b5bc354be WatchSource:0}: Error finding container a48bc15d0eaff812eeb69777395ed09dc1fc05ca551e364418939d4b5bc354be: Status 404 returned error can't find the container with id a48bc15d0eaff812eeb69777395ed09dc1fc05ca551e364418939d4b5bc354be
Sep 15 09:28:07.476166 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.476073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk" event={"ID":"3f8548bb-41f7-4e15-81ef-019fc4e1b93b","Type":"ContainerStarted","Data":"a48bc15d0eaff812eeb69777395ed09dc1fc05ca551e364418939d4b5bc354be"}
Sep 15 09:28:07.608159 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.608127    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"]
Sep 15 09:28:07.611919 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.611888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.614115 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.614089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c6646b4b-serving-cert\""
Sep 15 09:28:07.623497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.623299    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"]
Sep 15 09:28:07.717619 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.717580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea8740-9456-4367-87ac-9d228f10fabc-openshift-service-ca\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.717774 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.717730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjzzs\" (UniqueName: \"kubernetes.io/projected/47ea8740-9456-4367-87ac-9d228f10fabc-kube-api-access-tjzzs\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.717826 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.717772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-maas-api-tls\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.717826 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.717790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-metrics-certs\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.818704 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.818571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-maas-api-tls\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.818704 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.818640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-metrics-certs\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.818704 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.818697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea8740-9456-4367-87ac-9d228f10fabc-openshift-service-ca\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.818968 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:28:07.818808    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-serving-cert: secret "maas-api-e2e-ait-c6646b4b-serving-cert" not found
Sep 15 09:28:07.818968 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:28:07.818914    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-maas-api-tls podName:47ea8740-9456-4367-87ac-9d228f10fabc nodeName:}" failed. No retries permitted until 2026-09-15 09:28:08.318880333 +0000 UTC m=+852.148672234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-maas-api-tls") pod "maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd" (UID: "47ea8740-9456-4367-87ac-9d228f10fabc") : secret "maas-api-e2e-ait-c6646b4b-serving-cert" not found
Sep 15 09:28:07.818968 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.818936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjzzs\" (UniqueName: \"kubernetes.io/projected/47ea8740-9456-4367-87ac-9d228f10fabc-kube-api-access-tjzzs\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.819545 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.819517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea8740-9456-4367-87ac-9d228f10fabc-openshift-service-ca\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.821567 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.821545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-metrics-certs\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:07.826773 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:07.826750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjzzs\" (UniqueName: \"kubernetes.io/projected/47ea8740-9456-4367-87ac-9d228f10fabc-kube-api-access-tjzzs\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:08.323450 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:08.323417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-maas-api-tls\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:08.325884 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:08.325857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-maas-api-tls\") pod \"maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:08.460629 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:08.460591    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:28:08.482922 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:08.482889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk" event={"ID":"3f8548bb-41f7-4e15-81ef-019fc4e1b93b","Type":"ContainerStarted","Data":"996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8"}
Sep 15 09:28:08.483084 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:08.483019    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:08.502531 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:08.502482    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk" podStartSLOduration=2.081021958 podStartE2EDuration="2.502463944s" totalImagesPullingTime="421.441986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:28:08.500559161 +0000 UTC m=+852.330351062" watchObservedRunningTime="2026-09-15 09:28:08.502463944 +0000 UTC m=+852.332255848"
Sep 15 09:28:08.525995 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:08.525965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:08.796360 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:08.796318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"]
Sep 15 09:28:08.798092 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:28:08.798060    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ea8740_9456_4367_87ac_9d228f10fabc.slice/crio-21e0ca9580ba33b9c98a2533e880efb6bed09fbb785ab06b49f70915cee58b6f WatchSource:0}: Error finding container 21e0ca9580ba33b9c98a2533e880efb6bed09fbb785ab06b49f70915cee58b6f: Status 404 returned error can't find the container with id 21e0ca9580ba33b9c98a2533e880efb6bed09fbb785ab06b49f70915cee58b6f
Sep 15 09:28:09.490007 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:09.489954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd" event={"ID":"47ea8740-9456-4367-87ac-9d228f10fabc","Type":"ContainerStarted","Data":"1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06"}
Sep 15 09:28:09.490188 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:09.490028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd" event={"ID":"47ea8740-9456-4367-87ac-9d228f10fabc","Type":"ContainerStarted","Data":"21e0ca9580ba33b9c98a2533e880efb6bed09fbb785ab06b49f70915cee58b6f"}
Sep 15 09:28:09.512513 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:09.512465    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd" podStartSLOduration=2.20011655 podStartE2EDuration="2.512450113s" totalImagesPullingTime="312.333563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:28:09.510492256 +0000 UTC m=+853.340284158" watchObservedRunningTime="2026-09-15 09:28:09.512450113 +0000 UTC m=+853.342242016"
Sep 15 09:28:10.498506 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:10.498469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:13.102429 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:13.102392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:28:14.495975 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.495941    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:28:14.619626 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.619591    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"]
Sep 15 09:28:14.624006 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.623979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.626103 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.626079    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0e280a-serving-cert\""
Sep 15 09:28:14.632521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.632498    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"]
Sep 15 09:28:14.683988 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.683964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.684120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.684003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c9hq\" (UniqueName: \"kubernetes.io/projected/444b23db-7252-4d5c-a339-c52a3c28015f-kube-api-access-9c9hq\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.684120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.684023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444b23db-7252-4d5c-a339-c52a3c28015f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.684120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.684038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-metrics-certs\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.784873 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.784800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.784873 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.784836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c9hq\" (UniqueName: \"kubernetes.io/projected/444b23db-7252-4d5c-a339-c52a3c28015f-kube-api-access-9c9hq\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.785067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.784855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444b23db-7252-4d5c-a339-c52a3c28015f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.785067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.784897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-metrics-certs\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.785603 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.785569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444b23db-7252-4d5c-a339-c52a3c28015f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.787206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.787182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.787327 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.787310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-metrics-certs\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.792448 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.792428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c9hq\" (UniqueName: \"kubernetes.io/projected/444b23db-7252-4d5c-a339-c52a3c28015f-kube-api-access-9c9hq\") pod \"maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:14.938067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:14.938037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:15.213705 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:15.213680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"]
Sep 15 09:28:15.216324 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:28:15.216288    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444b23db_7252_4d5c_a339_c52a3c28015f.slice/crio-0fe36da74ef7ef0e1d0571b821b43092398bf9c4bfb3a7e47bb0f76e220b9d15 WatchSource:0}: Error finding container 0fe36da74ef7ef0e1d0571b821b43092398bf9c4bfb3a7e47bb0f76e220b9d15: Status 404 returned error can't find the container with id 0fe36da74ef7ef0e1d0571b821b43092398bf9c4bfb3a7e47bb0f76e220b9d15
Sep 15 09:28:15.527907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:15.527815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94" event={"ID":"444b23db-7252-4d5c-a339-c52a3c28015f","Type":"ContainerStarted","Data":"0fe36da74ef7ef0e1d0571b821b43092398bf9c4bfb3a7e47bb0f76e220b9d15"}
Sep 15 09:28:16.509419 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:16.509388    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:16.536168 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:16.536127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94" event={"ID":"444b23db-7252-4d5c-a339-c52a3c28015f","Type":"ContainerStarted","Data":"230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216"}
Sep 15 09:28:16.536603 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:16.536284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:16.557813 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:16.557758    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94" podStartSLOduration=2.148245012 podStartE2EDuration="2.557738533s" totalImagesPullingTime="409.493521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:28:16.556525694 +0000 UTC m=+860.386317599" watchObservedRunningTime="2026-09-15 09:28:16.557738533 +0000 UTC m=+860.387530437"
Sep 15 09:28:22.546660 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:22.546630    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:28:34.696915 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:34.696881    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:28:44.685635 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:44.685598    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"]
Sep 15 09:28:44.686162 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:44.685793    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd" podUID="47ea8740-9456-4367-87ac-9d228f10fabc" containerName="maas-api" containerID="cri-o://1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06" gracePeriod=30
Sep 15 09:28:44.927824 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:44.927803    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:45.059957 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.059861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-maas-api-tls\") pod \"47ea8740-9456-4367-87ac-9d228f10fabc\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") "
Sep 15 09:28:45.060111 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.059998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjzzs\" (UniqueName: \"kubernetes.io/projected/47ea8740-9456-4367-87ac-9d228f10fabc-kube-api-access-tjzzs\") pod \"47ea8740-9456-4367-87ac-9d228f10fabc\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") "
Sep 15 09:28:45.060111 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.060037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-metrics-certs\") pod \"47ea8740-9456-4367-87ac-9d228f10fabc\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") "
Sep 15 09:28:45.060111 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.060067    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea8740-9456-4367-87ac-9d228f10fabc-openshift-service-ca\") pod \"47ea8740-9456-4367-87ac-9d228f10fabc\" (UID: \"47ea8740-9456-4367-87ac-9d228f10fabc\") "
Sep 15 09:28:45.060700 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.060667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ea8740-9456-4367-87ac-9d228f10fabc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47ea8740-9456-4367-87ac-9d228f10fabc" (UID: "47ea8740-9456-4367-87ac-9d228f10fabc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:28:45.063026 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.062998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47ea8740-9456-4367-87ac-9d228f10fabc" (UID: "47ea8740-9456-4367-87ac-9d228f10fabc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:28:45.063106 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.063061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "47ea8740-9456-4367-87ac-9d228f10fabc" (UID: "47ea8740-9456-4367-87ac-9d228f10fabc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:28:45.063106 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.063074    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ea8740-9456-4367-87ac-9d228f10fabc-kube-api-access-tjzzs" (OuterVolumeSpecName: "kube-api-access-tjzzs") pod "47ea8740-9456-4367-87ac-9d228f10fabc" (UID: "47ea8740-9456-4367-87ac-9d228f10fabc"). InnerVolumeSpecName "kube-api-access-tjzzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:28:45.161443 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.161411    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjzzs\" (UniqueName: \"kubernetes.io/projected/47ea8740-9456-4367-87ac-9d228f10fabc-kube-api-access-tjzzs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:28:45.161443 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.161437    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:28:45.161443 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.161446    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ea8740-9456-4367-87ac-9d228f10fabc-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:28:45.161640 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.161456    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ea8740-9456-4367-87ac-9d228f10fabc-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:28:45.686463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.686427    2579 generic.go:356] "Generic (PLEG): container finished" podID="47ea8740-9456-4367-87ac-9d228f10fabc" containerID="1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06" exitCode=0
Sep 15 09:28:45.686908 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.686477    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"
Sep 15 09:28:45.686908 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.686494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd" event={"ID":"47ea8740-9456-4367-87ac-9d228f10fabc","Type":"ContainerDied","Data":"1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06"}
Sep 15 09:28:45.686908 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.686543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd" event={"ID":"47ea8740-9456-4367-87ac-9d228f10fabc","Type":"ContainerDied","Data":"21e0ca9580ba33b9c98a2533e880efb6bed09fbb785ab06b49f70915cee58b6f"}
Sep 15 09:28:45.686908 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.686573    2579 scope.go:117] "RemoveContainer" containerID="1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06"
Sep 15 09:28:45.697027 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.697007    2579 scope.go:117] "RemoveContainer" containerID="1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06"
Sep 15 09:28:45.697304 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:28:45.697284    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06\": container with ID starting with 1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06 not found: ID does not exist" containerID="1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06"
Sep 15 09:28:45.697389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.697313    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06"} err="failed to get container status \"1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06\": rpc error: code = NotFound desc = could not find container \"1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06\": container with ID starting with 1f1fa8f00ac316018ef25a59eb5c3e361f8e8106d2b4340eeaf56c53b6876f06 not found: ID does not exist"
Sep 15 09:28:45.710876 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.710844    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"]
Sep 15 09:28:45.716662 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:45.716641    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6646b4b-96889bccc-6wjzd"]
Sep 15 09:28:46.675063 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:46.675029    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ea8740-9456-4367-87ac-9d228f10fabc" path="/var/lib/kubelet/pods/47ea8740-9456-4367-87ac-9d228f10fabc/volumes"
Sep 15 09:28:50.088886 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.088799    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"]
Sep 15 09:28:50.089415 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.089286    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ea8740-9456-4367-87ac-9d228f10fabc" containerName="maas-api"
Sep 15 09:28:50.089415 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.089316    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ea8740-9456-4367-87ac-9d228f10fabc" containerName="maas-api"
Sep 15 09:28:50.089561 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.089481    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ea8740-9456-4367-87ac-9d228f10fabc" containerName="maas-api"
Sep 15 09:28:50.094026 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.094004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.096328 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.096305    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-357b9aa6-serving-cert\""
Sep 15 09:28:50.106359 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.106326    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"]
Sep 15 09:28:50.203525 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.203498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-maas-api-tls\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.203525 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.203528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-metrics-certs\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.203676 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.203548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc582563-3fef-449d-b170-a8a8c892b89b-openshift-service-ca\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.203676 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.203609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhg5q\" (UniqueName: \"kubernetes.io/projected/bc582563-3fef-449d-b170-a8a8c892b89b-kube-api-access-qhg5q\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.305187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.305160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-maas-api-tls\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.305187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.305189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-metrics-certs\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.305330 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.305208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc582563-3fef-449d-b170-a8a8c892b89b-openshift-service-ca\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.305330 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.305240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhg5q\" (UniqueName: \"kubernetes.io/projected/bc582563-3fef-449d-b170-a8a8c892b89b-kube-api-access-qhg5q\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.305910 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.305886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc582563-3fef-449d-b170-a8a8c892b89b-openshift-service-ca\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.307677 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.307657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-maas-api-tls\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.307764 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.307703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-metrics-certs\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.312157 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.312140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhg5q\" (UniqueName: \"kubernetes.io/projected/bc582563-3fef-449d-b170-a8a8c892b89b-kube-api-access-qhg5q\") pod \"maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.403490 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.403431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:28:50.406951 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.406937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:50.678498 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.678476    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"]
Sep 15 09:28:50.680727 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:28:50.680697    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc582563_3fef_449d_b170_a8a8c892b89b.slice/crio-c05ea4b50b82a528bbb8c271179ed36483cebabc283b72262dd6329832387a24 WatchSource:0}: Error finding container c05ea4b50b82a528bbb8c271179ed36483cebabc283b72262dd6329832387a24: Status 404 returned error can't find the container with id c05ea4b50b82a528bbb8c271179ed36483cebabc283b72262dd6329832387a24
Sep 15 09:28:50.713602 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:50.713574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc" event={"ID":"bc582563-3fef-449d-b170-a8a8c892b89b","Type":"ContainerStarted","Data":"c05ea4b50b82a528bbb8c271179ed36483cebabc283b72262dd6329832387a24"}
Sep 15 09:28:51.719539 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:51.719500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc" event={"ID":"bc582563-3fef-449d-b170-a8a8c892b89b","Type":"ContainerStarted","Data":"084b0c01859454ae789857a5a07b0978915cd5ca240fc240cacc247d5e29369f"}
Sep 15 09:28:51.719878 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:51.719627    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:28:51.736194 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:51.736155    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc" podStartSLOduration=1.430223018 podStartE2EDuration="1.736141591s" totalImagesPullingTime="305.918573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:28:51.735242225 +0000 UTC m=+895.565034139" watchObservedRunningTime="2026-09-15 09:28:51.736141591 +0000 UTC m=+895.565933493"
Sep 15 09:28:57.728954 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:28:57.728923    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:29:04.903748 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:04.903714    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:29:20.678289 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:20.678253    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"]
Sep 15 09:29:20.678915 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:20.678505    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc" podUID="bc582563-3fef-449d-b170-a8a8c892b89b" containerName="maas-api" containerID="cri-o://084b0c01859454ae789857a5a07b0978915cd5ca240fc240cacc247d5e29369f" gracePeriod=30
Sep 15 09:29:20.859021 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:20.858994    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc582563-3fef-449d-b170-a8a8c892b89b" containerID="084b0c01859454ae789857a5a07b0978915cd5ca240fc240cacc247d5e29369f" exitCode=0
Sep 15 09:29:20.859156 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:20.859067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc" event={"ID":"bc582563-3fef-449d-b170-a8a8c892b89b","Type":"ContainerDied","Data":"084b0c01859454ae789857a5a07b0978915cd5ca240fc240cacc247d5e29369f"}
Sep 15 09:29:20.925175 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:20.925152    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:29:21.059735 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.059655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhg5q\" (UniqueName: \"kubernetes.io/projected/bc582563-3fef-449d-b170-a8a8c892b89b-kube-api-access-qhg5q\") pod \"bc582563-3fef-449d-b170-a8a8c892b89b\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") "
Sep 15 09:29:21.059735 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.059710    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-maas-api-tls\") pod \"bc582563-3fef-449d-b170-a8a8c892b89b\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") "
Sep 15 09:29:21.059735 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.059733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-metrics-certs\") pod \"bc582563-3fef-449d-b170-a8a8c892b89b\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") "
Sep 15 09:29:21.059948 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.059756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc582563-3fef-449d-b170-a8a8c892b89b-openshift-service-ca\") pod \"bc582563-3fef-449d-b170-a8a8c892b89b\" (UID: \"bc582563-3fef-449d-b170-a8a8c892b89b\") "
Sep 15 09:29:21.060166 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.060137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc582563-3fef-449d-b170-a8a8c892b89b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc582563-3fef-449d-b170-a8a8c892b89b" (UID: "bc582563-3fef-449d-b170-a8a8c892b89b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:29:21.061800 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.061775    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc582563-3fef-449d-b170-a8a8c892b89b" (UID: "bc582563-3fef-449d-b170-a8a8c892b89b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:29:21.062270 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.062244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bc582563-3fef-449d-b170-a8a8c892b89b" (UID: "bc582563-3fef-449d-b170-a8a8c892b89b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:29:21.062270 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.062259    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc582563-3fef-449d-b170-a8a8c892b89b-kube-api-access-qhg5q" (OuterVolumeSpecName: "kube-api-access-qhg5q") pod "bc582563-3fef-449d-b170-a8a8c892b89b" (UID: "bc582563-3fef-449d-b170-a8a8c892b89b"). InnerVolumeSpecName "kube-api-access-qhg5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:29:21.161034 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.161005    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhg5q\" (UniqueName: \"kubernetes.io/projected/bc582563-3fef-449d-b170-a8a8c892b89b-kube-api-access-qhg5q\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:29:21.161034 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.161031    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:29:21.161174 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.161040    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc582563-3fef-449d-b170-a8a8c892b89b-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:29:21.161174 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.161048    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc582563-3fef-449d-b170-a8a8c892b89b-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:29:21.865877 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.865845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc" event={"ID":"bc582563-3fef-449d-b170-a8a8c892b89b","Type":"ContainerDied","Data":"c05ea4b50b82a528bbb8c271179ed36483cebabc283b72262dd6329832387a24"}
Sep 15 09:29:21.866273 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.865904    2579 scope.go:117] "RemoveContainer" containerID="084b0c01859454ae789857a5a07b0978915cd5ca240fc240cacc247d5e29369f"
Sep 15 09:29:21.866273 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.865853    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"
Sep 15 09:29:21.885783 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.885761    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"]
Sep 15 09:29:21.889596 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:21.889575    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-357b9aa6-dc88fc7bc-zd8cc"]
Sep 15 09:29:22.675193 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:22.675156    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc582563-3fef-449d-b170-a8a8c892b89b" path="/var/lib/kubelet/pods/bc582563-3fef-449d-b170-a8a8c892b89b/volumes"
Sep 15 09:29:25.791427 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.791390    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"]
Sep 15 09:29:25.791996 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.791938    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc582563-3fef-449d-b170-a8a8c892b89b" containerName="maas-api"
Sep 15 09:29:25.791996 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.791955    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc582563-3fef-449d-b170-a8a8c892b89b" containerName="maas-api"
Sep 15 09:29:25.792132 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.792020    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc582563-3fef-449d-b170-a8a8c892b89b" containerName="maas-api"
Sep 15 09:29:25.795015 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.794996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:25.797190 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.797167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3494740c-serving-cert\""
Sep 15 09:29:25.805916 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.805896    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"]
Sep 15 09:29:25.898244 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.898210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8txz\" (UniqueName: \"kubernetes.io/projected/87dd7c8d-3949-4a47-896f-e0affcadbd12-kube-api-access-k8txz\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:25.898244 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.898247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-metrics-certs\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:25.898467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.898268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-maas-api-tls\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:25.898467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.898368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd7c8d-3949-4a47-896f-e0affcadbd12-openshift-service-ca\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:25.998827 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.998792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd7c8d-3949-4a47-896f-e0affcadbd12-openshift-service-ca\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:25.998976 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.998878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8txz\" (UniqueName: \"kubernetes.io/projected/87dd7c8d-3949-4a47-896f-e0affcadbd12-kube-api-access-k8txz\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:25.998976 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.998900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-metrics-certs\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:25.998976 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.998916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-maas-api-tls\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:25.999501 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:25.999477    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd7c8d-3949-4a47-896f-e0affcadbd12-openshift-service-ca\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:26.001441 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:26.001417    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-maas-api-tls\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:26.001524 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:26.001451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-metrics-certs\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:26.005893 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:26.005867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8txz\" (UniqueName: \"kubernetes.io/projected/87dd7c8d-3949-4a47-896f-e0affcadbd12-kube-api-access-k8txz\") pod \"maas-api-e2e-ait-3494740c-649ff8f458-44csb\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:26.109578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:26.109519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:26.381245 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:26.381178    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"]
Sep 15 09:29:26.384326 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:29:26.384300    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dd7c8d_3949_4a47_896f_e0affcadbd12.slice/crio-127f0748f1a41ad9a2beedce2f18280a2f3ef758b0bced1e89fcf07ad80575ea WatchSource:0}: Error finding container 127f0748f1a41ad9a2beedce2f18280a2f3ef758b0bced1e89fcf07ad80575ea: Status 404 returned error can't find the container with id 127f0748f1a41ad9a2beedce2f18280a2f3ef758b0bced1e89fcf07ad80575ea
Sep 15 09:29:26.893493 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:26.893457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb" event={"ID":"87dd7c8d-3949-4a47-896f-e0affcadbd12","Type":"ContainerStarted","Data":"127f0748f1a41ad9a2beedce2f18280a2f3ef758b0bced1e89fcf07ad80575ea"}
Sep 15 09:29:27.104184 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:27.104147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:29:27.900758 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:27.900724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb" event={"ID":"87dd7c8d-3949-4a47-896f-e0affcadbd12","Type":"ContainerStarted","Data":"8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875"}
Sep 15 09:29:27.901142 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:27.900809    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:27.916529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:27.916478    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb" podStartSLOduration=2.338002413 podStartE2EDuration="2.9164628s" totalImagesPullingTime="578.460387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:29:27.915624608 +0000 UTC m=+931.745416512" watchObservedRunningTime="2026-09-15 09:29:27.9164628 +0000 UTC m=+931.746254702"
Sep 15 09:29:33.001985 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.001952    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"]
Sep 15 09:29:33.005815 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.005794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.008747 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.008724    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c183a5\"/\"kube-root-ca.crt\""
Sep 15 09:29:33.008872 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.008745    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c183a5\"/\"openshift-service-ca.crt\""
Sep 15 09:29:33.008872 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.008756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c183a5\"/\"simulator-w4-c183a5-kserve-self-signed-certs\""
Sep 15 09:29:33.008872 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.008743    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c183a5\"/\"default-dockercfg-k7hvd\""
Sep 15 09:29:33.012931 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.012905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"]
Sep 15 09:29:33.160725 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.160691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-tmp-dir\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.160856 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.160751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-model-cache\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.160856 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.160805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b9b9e8-2c21-4567-8269-365fa836fb82-tls-certs\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.160856 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.160824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-home\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.160856 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.160841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssxh\" (UniqueName: \"kubernetes.io/projected/94b9b9e8-2c21-4567-8269-365fa836fb82-kube-api-access-lssxh\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.160992 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.160896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-dshm\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.261945 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.261883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-tmp-dir\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.261945 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.261930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-model-cache\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.262134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.261954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b9b9e8-2c21-4567-8269-365fa836fb82-tls-certs\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.262134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.261974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-home\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.262134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.261990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssxh\" (UniqueName: \"kubernetes.io/projected/94b9b9e8-2c21-4567-8269-365fa836fb82-kube-api-access-lssxh\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.262134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.262042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-dshm\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.262379 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.262315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-tmp-dir\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.262379 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.262365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-model-cache\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.262469 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.262403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-home\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.264156 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.264138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-dshm\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.264440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.264423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b9b9e8-2c21-4567-8269-365fa836fb82-tls-certs\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.268625 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.268604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssxh\" (UniqueName: \"kubernetes.io/projected/94b9b9e8-2c21-4567-8269-365fa836fb82-kube-api-access-lssxh\") pod \"simulator-w4-c183a5-kserve-584456798d-vhllr\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") " pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.316630 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.316604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:33.585217 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.585192    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"]
Sep 15 09:29:33.586782 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:29:33.586751    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94b9b9e8_2c21_4567_8269_365fa836fb82.slice/crio-103a7d994ba60a996e356b70e6483d638fe70f0d4d633a3f9882e780189bf719 WatchSource:0}: Error finding container 103a7d994ba60a996e356b70e6483d638fe70f0d4d633a3f9882e780189bf719: Status 404 returned error can't find the container with id 103a7d994ba60a996e356b70e6483d638fe70f0d4d633a3f9882e780189bf719
Sep 15 09:29:33.911399 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.911375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:33.932760 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.932729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr" event={"ID":"94b9b9e8-2c21-4567-8269-365fa836fb82","Type":"ContainerStarted","Data":"9f5fc07187939af66a3972033195045bcc019407764fa505151f450d7dc60999"}
Sep 15 09:29:33.932897 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.932764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr" event={"ID":"94b9b9e8-2c21-4567-8269-365fa836fb82","Type":"ContainerStarted","Data":"103a7d994ba60a996e356b70e6483d638fe70f0d4d633a3f9882e780189bf719"}
Sep 15 09:29:33.950569 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:33.950506    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr" podStartSLOduration=1.950486208 podStartE2EDuration="1.950486208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:29:33.949351696 +0000 UTC m=+937.779143596" watchObservedRunningTime="2026-09-15 09:29:33.950486208 +0000 UTC m=+937.780278114"
Sep 15 09:29:43.316967 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:43.316925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:43.317377 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:43.316981    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:43.329669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:43.329642    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:43.992013 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:43.991983    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:29:47.104532 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:47.104497    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:29:55.548219 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.548181    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"]
Sep 15 09:29:55.548723 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.548499    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb" podUID="87dd7c8d-3949-4a47-896f-e0affcadbd12" containerName="maas-api" containerID="cri-o://8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875" gracePeriod=30
Sep 15 09:29:55.794140 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.794119    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:55.970239 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.970207    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd7c8d-3949-4a47-896f-e0affcadbd12-openshift-service-ca\") pod \"87dd7c8d-3949-4a47-896f-e0affcadbd12\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") "
Sep 15 09:29:55.970411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.970290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-metrics-certs\") pod \"87dd7c8d-3949-4a47-896f-e0affcadbd12\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") "
Sep 15 09:29:55.970411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.970372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8txz\" (UniqueName: \"kubernetes.io/projected/87dd7c8d-3949-4a47-896f-e0affcadbd12-kube-api-access-k8txz\") pod \"87dd7c8d-3949-4a47-896f-e0affcadbd12\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") "
Sep 15 09:29:55.970411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.970395    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-maas-api-tls\") pod \"87dd7c8d-3949-4a47-896f-e0affcadbd12\" (UID: \"87dd7c8d-3949-4a47-896f-e0affcadbd12\") "
Sep 15 09:29:55.970620 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.970597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87dd7c8d-3949-4a47-896f-e0affcadbd12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87dd7c8d-3949-4a47-896f-e0affcadbd12" (UID: "87dd7c8d-3949-4a47-896f-e0affcadbd12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:29:55.970778 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.970764    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd7c8d-3949-4a47-896f-e0affcadbd12-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:29:55.972509 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.972488    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "87dd7c8d-3949-4a47-896f-e0affcadbd12" (UID: "87dd7c8d-3949-4a47-896f-e0affcadbd12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:29:55.972613 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.972509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87dd7c8d-3949-4a47-896f-e0affcadbd12" (UID: "87dd7c8d-3949-4a47-896f-e0affcadbd12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:29:55.972613 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:55.972520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dd7c8d-3949-4a47-896f-e0affcadbd12-kube-api-access-k8txz" (OuterVolumeSpecName: "kube-api-access-k8txz") pod "87dd7c8d-3949-4a47-896f-e0affcadbd12" (UID: "87dd7c8d-3949-4a47-896f-e0affcadbd12"). InnerVolumeSpecName "kube-api-access-k8txz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:29:56.042647 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.042614    2579 generic.go:356] "Generic (PLEG): container finished" podID="87dd7c8d-3949-4a47-896f-e0affcadbd12" containerID="8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875" exitCode=0
Sep 15 09:29:56.042767 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.042666    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"
Sep 15 09:29:56.042767 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.042705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb" event={"ID":"87dd7c8d-3949-4a47-896f-e0affcadbd12","Type":"ContainerDied","Data":"8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875"}
Sep 15 09:29:56.042882 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.042772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb" event={"ID":"87dd7c8d-3949-4a47-896f-e0affcadbd12","Type":"ContainerDied","Data":"127f0748f1a41ad9a2beedce2f18280a2f3ef758b0bced1e89fcf07ad80575ea"}
Sep 15 09:29:56.042882 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.042806    2579 scope.go:117] "RemoveContainer" containerID="8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875"
Sep 15 09:29:56.051526 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.051508    2579 scope.go:117] "RemoveContainer" containerID="8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875"
Sep 15 09:29:56.051800 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:29:56.051780    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875\": container with ID starting with 8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875 not found: ID does not exist" containerID="8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875"
Sep 15 09:29:56.051869 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.051807    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875"} err="failed to get container status \"8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875\": rpc error: code = NotFound desc = could not find container \"8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875\": container with ID starting with 8e4a70819d1d28452bd9bbb5ff682b81a25e926520c11a2c05b87d0d70eba875 not found: ID does not exist"
Sep 15 09:29:56.065255 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.065234    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"]
Sep 15 09:29:56.067035 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.067017    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3494740c-649ff8f458-44csb"]
Sep 15 09:29:56.072049 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.072020    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8txz\" (UniqueName: \"kubernetes.io/projected/87dd7c8d-3949-4a47-896f-e0affcadbd12-kube-api-access-k8txz\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:29:56.072049 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.072049    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:29:56.072169 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.072062    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd7c8d-3949-4a47-896f-e0affcadbd12-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:29:56.675053 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:29:56.675023    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87dd7c8d-3949-4a47-896f-e0affcadbd12" path="/var/lib/kubelet/pods/87dd7c8d-3949-4a47-896f-e0affcadbd12/volumes"
Sep 15 09:30:03.498691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:03.498660    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:30:04.189717 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.189682    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"]
Sep 15 09:30:04.190099 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.190086    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87dd7c8d-3949-4a47-896f-e0affcadbd12" containerName="maas-api"
Sep 15 09:30:04.190099 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.190100    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="87dd7c8d-3949-4a47-896f-e0affcadbd12" containerName="maas-api"
Sep 15 09:30:04.190198 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.190173    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="87dd7c8d-3949-4a47-896f-e0affcadbd12" containerName="maas-api"
Sep 15 09:30:04.194486 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.194463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.196786 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.196763    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e76bac\"/\"openshift-service-ca.crt\""
Sep 15 09:30:04.197603 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.197586    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e76bac\"/\"kube-root-ca.crt\""
Sep 15 09:30:04.197699 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.197639    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e76bac\"/\"default-dockercfg-sjjrz\""
Sep 15 09:30:04.197699 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.197674    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e76bac\"/\"simulator-w0-e76bac-kserve-self-signed-certs\""
Sep 15 09:30:04.200753 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.200733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"]
Sep 15 09:30:04.236968 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.236948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-dshm\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.237085 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.237007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-model-cache\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.237085 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.237049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-home\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.237199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.237110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-tmp-dir\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.237199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.237135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszzb\" (UniqueName: \"kubernetes.io/projected/595eaf3a-aaaf-4762-870e-0a70d4838636-kube-api-access-wszzb\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.237275 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.237226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595eaf3a-aaaf-4762-870e-0a70d4838636-tls-certs\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.338259 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.338227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595eaf3a-aaaf-4762-870e-0a70d4838636-tls-certs\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.338450 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.338275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-dshm\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.338450 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.338401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-model-cache\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.338450 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.338441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-home\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.338663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.338491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-tmp-dir\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.338663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.338508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszzb\" (UniqueName: \"kubernetes.io/projected/595eaf3a-aaaf-4762-870e-0a70d4838636-kube-api-access-wszzb\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.338798 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.338774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-model-cache\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.338859 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.338807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-home\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.338859 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.338829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-tmp-dir\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.340595 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.340573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-dshm\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.340819 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.340801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595eaf3a-aaaf-4762-870e-0a70d4838636-tls-certs\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.350158 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.350138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszzb\" (UniqueName: \"kubernetes.io/projected/595eaf3a-aaaf-4762-870e-0a70d4838636-kube-api-access-wszzb\") pod \"simulator-w0-e76bac-kserve-7df47b4b57-rg5lc\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") " pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.504863 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.504807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:04.773788 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:04.773757    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"]
Sep 15 09:30:04.776232 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:30:04.776205    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod595eaf3a_aaaf_4762_870e_0a70d4838636.slice/crio-b4e67dda3263da52ee83e05e2e170edffda00c21cd6d0aac9a6afe97b0c33619 WatchSource:0}: Error finding container b4e67dda3263da52ee83e05e2e170edffda00c21cd6d0aac9a6afe97b0c33619: Status 404 returned error can't find the container with id b4e67dda3263da52ee83e05e2e170edffda00c21cd6d0aac9a6afe97b0c33619
Sep 15 09:30:05.102454 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.102371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc" event={"ID":"595eaf3a-aaaf-4762-870e-0a70d4838636","Type":"ContainerStarted","Data":"06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e"}
Sep 15 09:30:05.102454 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.102412    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc" event={"ID":"595eaf3a-aaaf-4762-870e-0a70d4838636","Type":"ContainerStarted","Data":"b4e67dda3263da52ee83e05e2e170edffda00c21cd6d0aac9a6afe97b0c33619"}
Sep 15 09:30:05.102629 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.102580    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"]
Sep 15 09:30:05.106063 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.106045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.107807 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.107791    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b75aaea1-serving-cert\""
Sep 15 09:30:05.112376 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.112354    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"]
Sep 15 09:30:05.146303 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.146273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-metrics-certs\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.146464 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.146357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwzj\" (UniqueName: \"kubernetes.io/projected/7af2583e-368a-4988-9ad4-a14eec7f6009-kube-api-access-8zwzj\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.146464 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.146436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-maas-api-tls\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.146586 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.146569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af2583e-368a-4988-9ad4-a14eec7f6009-openshift-service-ca\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.153351 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.153297    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc" podStartSLOduration=1.153283565 podStartE2EDuration="1.153283565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:30:05.137943897 +0000 UTC m=+968.967735800" watchObservedRunningTime="2026-09-15 09:30:05.153283565 +0000 UTC m=+968.983075467"
Sep 15 09:30:05.247842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.247814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-maas-api-tls\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.247974 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.247893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af2583e-368a-4988-9ad4-a14eec7f6009-openshift-service-ca\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.247974 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:30:05.247920    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-serving-cert: secret "maas-api-e2e-derive-b75aaea1-serving-cert" not found
Sep 15 09:30:05.247974 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.247942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-metrics-certs\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.247974 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.247957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwzj\" (UniqueName: \"kubernetes.io/projected/7af2583e-368a-4988-9ad4-a14eec7f6009-kube-api-access-8zwzj\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.248154 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:30:05.247986    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-maas-api-tls podName:7af2583e-368a-4988-9ad4-a14eec7f6009 nodeName:}" failed. No retries permitted until 2026-09-15 09:30:05.747970543 +0000 UTC m=+969.577762423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-maas-api-tls") pod "maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w" (UID: "7af2583e-368a-4988-9ad4-a14eec7f6009") : secret "maas-api-e2e-derive-b75aaea1-serving-cert" not found
Sep 15 09:30:05.248616 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.248589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af2583e-368a-4988-9ad4-a14eec7f6009-openshift-service-ca\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.250314 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.250296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-metrics-certs\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.258023 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.257999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwzj\" (UniqueName: \"kubernetes.io/projected/7af2583e-368a-4988-9ad4-a14eec7f6009-kube-api-access-8zwzj\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.754088 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.754052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-maas-api-tls\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:05.756448 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:05.756423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-maas-api-tls\") pod \"maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:06.039383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:06.039296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:06.313635 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:06.313609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"]
Sep 15 09:30:06.315391 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:30:06.315362    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af2583e_368a_4988_9ad4_a14eec7f6009.slice/crio-830e8ce0bd1efcfaf389b4451d24811f50e10563ed027b5e173f48ba1c2e38a6 WatchSource:0}: Error finding container 830e8ce0bd1efcfaf389b4451d24811f50e10563ed027b5e173f48ba1c2e38a6: Status 404 returned error can't find the container with id 830e8ce0bd1efcfaf389b4451d24811f50e10563ed027b5e173f48ba1c2e38a6
Sep 15 09:30:07.114279 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:07.114190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w" event={"ID":"7af2583e-368a-4988-9ad4-a14eec7f6009","Type":"ContainerStarted","Data":"8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593"}
Sep 15 09:30:07.114279 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:07.114240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w" event={"ID":"7af2583e-368a-4988-9ad4-a14eec7f6009","Type":"ContainerStarted","Data":"830e8ce0bd1efcfaf389b4451d24811f50e10563ed027b5e173f48ba1c2e38a6"}
Sep 15 09:30:07.114717 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:07.114357    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:07.129554 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:07.129501    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w" podStartSLOduration=1.657851988 podStartE2EDuration="2.129483991s" totalImagesPullingTime="471.632003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:30:07.128812609 +0000 UTC m=+970.958604513" watchObservedRunningTime="2026-09-15 09:30:07.129483991 +0000 UTC m=+970.959275895"
Sep 15 09:30:13.124668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:13.124639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:14.505269 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:14.505237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:14.505671 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:14.505371    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:14.517709 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:14.517688    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:15.166388 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:15.166360    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:30:20.910318 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:20.910239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:30:25.094952 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.094919    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"]
Sep 15 09:30:25.098128 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.098110    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.101384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.101332    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3b0480\"/\"kube-root-ca.crt\""
Sep 15 09:30:25.101503 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.101389    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3b0480\"/\"openshift-service-ca.crt\""
Sep 15 09:30:25.101503 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.101406    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3b0480\"/\"autores-model-3b0480-kserve-self-signed-certs\""
Sep 15 09:30:25.101503 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.101403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3b0480\"/\"default-dockercfg-f9h56\""
Sep 15 09:30:25.108999 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.108977    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"]
Sep 15 09:30:25.227860 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.227824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sh4\" (UniqueName: \"kubernetes.io/projected/ee930c43-571d-4b04-861c-f1bf04671313-kube-api-access-f8sh4\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.228024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.227881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee930c43-571d-4b04-861c-f1bf04671313-tls-certs\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.228024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.227946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-model-cache\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.228024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.227971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-home\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.228024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.227988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-tmp-dir\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.228221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.228089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-dshm\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.329129 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.329101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-dshm\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.329359 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.329151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sh4\" (UniqueName: \"kubernetes.io/projected/ee930c43-571d-4b04-861c-f1bf04671313-kube-api-access-f8sh4\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.329642 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.329615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee930c43-571d-4b04-861c-f1bf04671313-tls-certs\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.329723 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.329681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-model-cache\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.329723 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.329713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-home\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.329823 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.329742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-tmp-dir\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.330183 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.330155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-tmp-dir\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.330367 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.330216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-model-cache\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.332396 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.332371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-dshm\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.333371 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.332860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-home\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.333371 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.333108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee930c43-571d-4b04-861c-f1bf04671313-tls-certs\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.337763 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.337463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sh4\" (UniqueName: \"kubernetes.io/projected/ee930c43-571d-4b04-861c-f1bf04671313-kube-api-access-f8sh4\") pod \"autores-model-3b0480-kserve-7cbd596759-qj8r8\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") " pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.408324 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.408301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:25.678889 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:25.678861    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"]
Sep 15 09:30:25.680654 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:30:25.680629    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee930c43_571d_4b04_861c_f1bf04671313.slice/crio-bbb2f0f2db08c41d273f8b583315a0cc60cfe9be59eede1a723295fbfb4cb9bb WatchSource:0}: Error finding container bbb2f0f2db08c41d273f8b583315a0cc60cfe9be59eede1a723295fbfb4cb9bb: Status 404 returned error can't find the container with id bbb2f0f2db08c41d273f8b583315a0cc60cfe9be59eede1a723295fbfb4cb9bb
Sep 15 09:30:26.211392 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:26.211359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8" event={"ID":"ee930c43-571d-4b04-861c-f1bf04671313","Type":"ContainerStarted","Data":"70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43"}
Sep 15 09:30:26.211392 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:26.211397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8" event={"ID":"ee930c43-571d-4b04-861c-f1bf04671313","Type":"ContainerStarted","Data":"bbb2f0f2db08c41d273f8b583315a0cc60cfe9be59eede1a723295fbfb4cb9bb"}
Sep 15 09:30:26.228263 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:26.228213    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8" podStartSLOduration=1.228197083 podStartE2EDuration="1.228197083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:30:26.225880932 +0000 UTC m=+990.055672835" watchObservedRunningTime="2026-09-15 09:30:26.228197083 +0000 UTC m=+990.057988983"
Sep 15 09:30:34.005307 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:34.005271    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:30:35.408523 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:35.408489    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:35.408881 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:35.408551    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:35.421645 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:35.421618    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:36.272731 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:36.272698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:30:42.460991 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.460955    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"]
Sep 15 09:30:42.461429 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.461226    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w" podUID="7af2583e-368a-4988-9ad4-a14eec7f6009" containerName="maas-api" containerID="cri-o://8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593" gracePeriod=30
Sep 15 09:30:42.705859 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.705835    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:42.891651 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.891579    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-metrics-certs\") pod \"7af2583e-368a-4988-9ad4-a14eec7f6009\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") "
Sep 15 09:30:42.891651 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.891623    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af2583e-368a-4988-9ad4-a14eec7f6009-openshift-service-ca\") pod \"7af2583e-368a-4988-9ad4-a14eec7f6009\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") "
Sep 15 09:30:42.891651 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.891653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-maas-api-tls\") pod \"7af2583e-368a-4988-9ad4-a14eec7f6009\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") "
Sep 15 09:30:42.891851 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.891704    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zwzj\" (UniqueName: \"kubernetes.io/projected/7af2583e-368a-4988-9ad4-a14eec7f6009-kube-api-access-8zwzj\") pod \"7af2583e-368a-4988-9ad4-a14eec7f6009\" (UID: \"7af2583e-368a-4988-9ad4-a14eec7f6009\") "
Sep 15 09:30:42.892023 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.891992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af2583e-368a-4988-9ad4-a14eec7f6009-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7af2583e-368a-4988-9ad4-a14eec7f6009" (UID: "7af2583e-368a-4988-9ad4-a14eec7f6009"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:30:42.893783 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.893757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7af2583e-368a-4988-9ad4-a14eec7f6009" (UID: "7af2583e-368a-4988-9ad4-a14eec7f6009"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:30:42.893888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.893805    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7af2583e-368a-4988-9ad4-a14eec7f6009" (UID: "7af2583e-368a-4988-9ad4-a14eec7f6009"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:30:42.893888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.893803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af2583e-368a-4988-9ad4-a14eec7f6009-kube-api-access-8zwzj" (OuterVolumeSpecName: "kube-api-access-8zwzj") pod "7af2583e-368a-4988-9ad4-a14eec7f6009" (UID: "7af2583e-368a-4988-9ad4-a14eec7f6009"). InnerVolumeSpecName "kube-api-access-8zwzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:30:42.992544 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.992518    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zwzj\" (UniqueName: \"kubernetes.io/projected/7af2583e-368a-4988-9ad4-a14eec7f6009-kube-api-access-8zwzj\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:30:42.992668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.992542    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:30:42.992668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.992563    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af2583e-368a-4988-9ad4-a14eec7f6009-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:30:42.992668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:42.992573    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af2583e-368a-4988-9ad4-a14eec7f6009-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:30:43.299735 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:43.299705    2579 generic.go:356] "Generic (PLEG): container finished" podID="7af2583e-368a-4988-9ad4-a14eec7f6009" containerID="8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593" exitCode=0
Sep 15 09:30:43.299907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:43.299746    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"
Sep 15 09:30:43.299907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:43.299789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w" event={"ID":"7af2583e-368a-4988-9ad4-a14eec7f6009","Type":"ContainerDied","Data":"8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593"}
Sep 15 09:30:43.299907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:43.299837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w" event={"ID":"7af2583e-368a-4988-9ad4-a14eec7f6009","Type":"ContainerDied","Data":"830e8ce0bd1efcfaf389b4451d24811f50e10563ed027b5e173f48ba1c2e38a6"}
Sep 15 09:30:43.299907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:43.299859    2579 scope.go:117] "RemoveContainer" containerID="8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593"
Sep 15 09:30:43.308966 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:43.308950    2579 scope.go:117] "RemoveContainer" containerID="8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593"
Sep 15 09:30:43.309239 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:30:43.309216    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593\": container with ID starting with 8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593 not found: ID does not exist" containerID="8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593"
Sep 15 09:30:43.309303 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:43.309250    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593"} err="failed to get container status \"8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593\": rpc error: code = NotFound desc = could not find container \"8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593\": container with ID starting with 8a944fb299e63bbc26266708965dbd20a85dbc4c6ef0b27ddcef8f352439f593 not found: ID does not exist"
Sep 15 09:30:43.320757 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:43.320734    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"]
Sep 15 09:30:43.324006 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:43.323987    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b75aaea1-7b8fc55f5d-brp4w"]
Sep 15 09:30:44.675230 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:44.675195    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af2583e-368a-4988-9ad4-a14eec7f6009" path="/var/lib/kubelet/pods/7af2583e-368a-4988-9ad4-a14eec7f6009/volumes"
Sep 15 09:30:55.194691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:55.194648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:30:57.104064 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:30:57.103953    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"]
Sep 15 09:31:07.718098 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:07.718055    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"]
Sep 15 09:31:07.718668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:07.718285    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk" podUID="3f8548bb-41f7-4e15-81ef-019fc4e1b93b" containerName="maas-api" containerID="cri-o://996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8" gracePeriod=30
Sep 15 09:31:07.981531 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:07.981456    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:31:08.131135 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.131092    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-maas-api-tls\") pod \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") "
Sep 15 09:31:08.131321 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.131193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9ltd\" (UniqueName: \"kubernetes.io/projected/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-kube-api-access-f9ltd\") pod \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") "
Sep 15 09:31:08.131321 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.131292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-openshift-service-ca\") pod \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") "
Sep 15 09:31:08.131430 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.131367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-metrics-certs\") pod \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\" (UID: \"3f8548bb-41f7-4e15-81ef-019fc4e1b93b\") "
Sep 15 09:31:08.131682 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.131651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f8548bb-41f7-4e15-81ef-019fc4e1b93b" (UID: "3f8548bb-41f7-4e15-81ef-019fc4e1b93b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:31:08.131806 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.131770    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:08.133565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.133540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f8548bb-41f7-4e15-81ef-019fc4e1b93b" (UID: "3f8548bb-41f7-4e15-81ef-019fc4e1b93b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:31:08.133850 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.133821    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-kube-api-access-f9ltd" (OuterVolumeSpecName: "kube-api-access-f9ltd") pod "3f8548bb-41f7-4e15-81ef-019fc4e1b93b" (UID: "3f8548bb-41f7-4e15-81ef-019fc4e1b93b"). InnerVolumeSpecName "kube-api-access-f9ltd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:31:08.133850 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.133822    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f8548bb-41f7-4e15-81ef-019fc4e1b93b" (UID: "3f8548bb-41f7-4e15-81ef-019fc4e1b93b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:31:08.232929 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.232827    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ltd\" (UniqueName: \"kubernetes.io/projected/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-kube-api-access-f9ltd\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:08.232929 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.232866    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:08.232929 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.232876    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8548bb-41f7-4e15-81ef-019fc4e1b93b-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:08.432784 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.432748    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f8548bb-41f7-4e15-81ef-019fc4e1b93b" containerID="996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8" exitCode=0
Sep 15 09:31:08.432963 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.432797    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"
Sep 15 09:31:08.432963 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.432825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk" event={"ID":"3f8548bb-41f7-4e15-81ef-019fc4e1b93b","Type":"ContainerDied","Data":"996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8"}
Sep 15 09:31:08.432963 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.432911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk" event={"ID":"3f8548bb-41f7-4e15-81ef-019fc4e1b93b","Type":"ContainerDied","Data":"a48bc15d0eaff812eeb69777395ed09dc1fc05ca551e364418939d4b5bc354be"}
Sep 15 09:31:08.432963 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.432944    2579 scope.go:117] "RemoveContainer" containerID="996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8"
Sep 15 09:31:08.443262 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.443243    2579 scope.go:117] "RemoveContainer" containerID="996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8"
Sep 15 09:31:08.443640 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:31:08.443621    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8\": container with ID starting with 996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8 not found: ID does not exist" containerID="996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8"
Sep 15 09:31:08.443693 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.443648    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8"} err="failed to get container status \"996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8\": rpc error: code = NotFound desc = could not find container \"996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8\": container with ID starting with 996f5689d4b225356efa226e5284da91b70ce23d0d5645d04a1d98e9180c7fb8 not found: ID does not exist"
Sep 15 09:31:08.454744 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.454706    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"]
Sep 15 09:31:08.456601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.456575    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3b0480-67d4b4f556-9p2sk"]
Sep 15 09:31:08.675957 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:08.675925    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f8548bb-41f7-4e15-81ef-019fc4e1b93b" path="/var/lib/kubelet/pods/3f8548bb-41f7-4e15-81ef-019fc4e1b93b/volumes"
Sep 15 09:31:10.394261 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:10.394225    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:31:12.124272 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.124211    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8" podUID="ee930c43-571d-4b04-861c-f1bf04671313" containerName="main" containerID="cri-o://70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43" gracePeriod=45
Sep 15 09:31:12.371724 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.371699    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:31:12.455173 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.455139    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee930c43-571d-4b04-861c-f1bf04671313" containerID="70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43" exitCode=0
Sep 15 09:31:12.455331 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.455179    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"
Sep 15 09:31:12.455331 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.455220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8" event={"ID":"ee930c43-571d-4b04-861c-f1bf04671313","Type":"ContainerDied","Data":"70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43"}
Sep 15 09:31:12.455331 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.455257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8" event={"ID":"ee930c43-571d-4b04-861c-f1bf04671313","Type":"ContainerDied","Data":"bbb2f0f2db08c41d273f8b583315a0cc60cfe9be59eede1a723295fbfb4cb9bb"}
Sep 15 09:31:12.455331 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.455275    2579 scope.go:117] "RemoveContainer" containerID="70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43"
Sep 15 09:31:12.463545 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.463526    2579 scope.go:117] "RemoveContainer" containerID="70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43"
Sep 15 09:31:12.463823 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:31:12.463804    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43\": container with ID starting with 70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43 not found: ID does not exist" containerID="70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43"
Sep 15 09:31:12.463877 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.463836    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43"} err="failed to get container status \"70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43\": rpc error: code = NotFound desc = could not find container \"70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43\": container with ID starting with 70a0cbc3177d8ce81d0b626ec43b4baec55655af3ca7f22a0ca6435e2e9b1c43 not found: ID does not exist"
Sep 15 09:31:12.473253 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473230    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-tmp-dir\") pod \"ee930c43-571d-4b04-861c-f1bf04671313\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") "
Sep 15 09:31:12.473387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee930c43-571d-4b04-861c-f1bf04671313-tls-certs\") pod \"ee930c43-571d-4b04-861c-f1bf04671313\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") "
Sep 15 09:31:12.473387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473355    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8sh4\" (UniqueName: \"kubernetes.io/projected/ee930c43-571d-4b04-861c-f1bf04671313-kube-api-access-f8sh4\") pod \"ee930c43-571d-4b04-861c-f1bf04671313\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") "
Sep 15 09:31:12.473387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-model-cache\") pod \"ee930c43-571d-4b04-861c-f1bf04671313\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") "
Sep 15 09:31:12.473548 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473502    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee930c43-571d-4b04-861c-f1bf04671313" (UID: "ee930c43-571d-4b04-861c-f1bf04671313"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:31:12.473548 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-home\") pod \"ee930c43-571d-4b04-861c-f1bf04671313\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") "
Sep 15 09:31:12.473652 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-dshm\") pod \"ee930c43-571d-4b04-861c-f1bf04671313\" (UID: \"ee930c43-571d-4b04-861c-f1bf04671313\") "
Sep 15 09:31:12.473702 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473565    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee930c43-571d-4b04-861c-f1bf04671313" (UID: "ee930c43-571d-4b04-861c-f1bf04671313"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:31:12.473750 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-home" (OuterVolumeSpecName: "home") pod "ee930c43-571d-4b04-861c-f1bf04671313" (UID: "ee930c43-571d-4b04-861c-f1bf04671313"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:31:12.473922 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473907    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:12.473983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473924    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:12.473983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.473949    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:12.475578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.475559    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-dshm" (OuterVolumeSpecName: "dshm") pod "ee930c43-571d-4b04-861c-f1bf04671313" (UID: "ee930c43-571d-4b04-861c-f1bf04671313"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:31:12.475852 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.475836    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee930c43-571d-4b04-861c-f1bf04671313-kube-api-access-f8sh4" (OuterVolumeSpecName: "kube-api-access-f8sh4") pod "ee930c43-571d-4b04-861c-f1bf04671313" (UID: "ee930c43-571d-4b04-861c-f1bf04671313"). InnerVolumeSpecName "kube-api-access-f8sh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:31:12.475931 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.475840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee930c43-571d-4b04-861c-f1bf04671313-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee930c43-571d-4b04-861c-f1bf04671313" (UID: "ee930c43-571d-4b04-861c-f1bf04671313"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:31:12.575295 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.575272    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee930c43-571d-4b04-861c-f1bf04671313-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:12.575295 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.575293    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8sh4\" (UniqueName: \"kubernetes.io/projected/ee930c43-571d-4b04-861c-f1bf04671313-kube-api-access-f8sh4\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:12.575469 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.575303    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee930c43-571d-4b04-861c-f1bf04671313-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:31:12.769912 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.769844    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"]
Sep 15 09:31:12.771660 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:12.771641    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-3b0480/autores-model-3b0480-kserve-7cbd596759-qj8r8"]
Sep 15 09:31:14.675459 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:14.675423    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee930c43-571d-4b04-861c-f1bf04671313" path="/var/lib/kubelet/pods/ee930c43-571d-4b04-861c-f1bf04671313/volumes"
Sep 15 09:31:21.838017 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.837983    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"]
Sep 15 09:31:21.838497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.838470    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8548bb-41f7-4e15-81ef-019fc4e1b93b" containerName="maas-api"
Sep 15 09:31:21.838497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.838493    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8548bb-41f7-4e15-81ef-019fc4e1b93b" containerName="maas-api"
Sep 15 09:31:21.838663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.838526    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af2583e-368a-4988-9ad4-a14eec7f6009" containerName="maas-api"
Sep 15 09:31:21.838663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.838536    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af2583e-368a-4988-9ad4-a14eec7f6009" containerName="maas-api"
Sep 15 09:31:21.838663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.838572    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee930c43-571d-4b04-861c-f1bf04671313" containerName="main"
Sep 15 09:31:21.838663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.838582    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee930c43-571d-4b04-861c-f1bf04671313" containerName="main"
Sep 15 09:31:21.838894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.838695    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee930c43-571d-4b04-861c-f1bf04671313" containerName="main"
Sep 15 09:31:21.838894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.838707    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af2583e-368a-4988-9ad4-a14eec7f6009" containerName="maas-api"
Sep 15 09:31:21.838894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.838718    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f8548bb-41f7-4e15-81ef-019fc4e1b93b" containerName="maas-api"
Sep 15 09:31:21.843386 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.843358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:21.845437 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.845414    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-5b654a-serving-cert\""
Sep 15 09:31:21.853507 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.853481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"]
Sep 15 09:31:21.965718 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.965683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-metrics-certs\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:21.965718 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.965721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-maas-api-tls\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:21.965964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.965752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nfv\" (UniqueName: \"kubernetes.io/projected/402cfbbd-e89a-41c4-96f4-5894064fcd5f-kube-api-access-w9nfv\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:21.965964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:21.965840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402cfbbd-e89a-41c4-96f4-5894064fcd5f-openshift-service-ca\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.067041 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.067005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402cfbbd-e89a-41c4-96f4-5894064fcd5f-openshift-service-ca\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.067206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.067090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-metrics-certs\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.067206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.067108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-maas-api-tls\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.067206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.067139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nfv\" (UniqueName: \"kubernetes.io/projected/402cfbbd-e89a-41c4-96f4-5894064fcd5f-kube-api-access-w9nfv\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.067685 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.067662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402cfbbd-e89a-41c4-96f4-5894064fcd5f-openshift-service-ca\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.069666 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.069637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-metrics-certs\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.069750 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.069672    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-maas-api-tls\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.075687 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.075666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nfv\" (UniqueName: \"kubernetes.io/projected/402cfbbd-e89a-41c4-96f4-5894064fcd5f-kube-api-access-w9nfv\") pod \"maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.156420 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.156324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:22.430610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.430555    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"]
Sep 15 09:31:22.433721 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:31:22.433658    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402cfbbd_e89a_41c4_96f4_5894064fcd5f.slice/crio-954e4a51ea3d62f86f399e8940758c38ab1d002739fe50b1a825721f28ff3407 WatchSource:0}: Error finding container 954e4a51ea3d62f86f399e8940758c38ab1d002739fe50b1a825721f28ff3407: Status 404 returned error can't find the container with id 954e4a51ea3d62f86f399e8940758c38ab1d002739fe50b1a825721f28ff3407
Sep 15 09:31:22.435581 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.435562    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:31:22.507346 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:22.507309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw" event={"ID":"402cfbbd-e89a-41c4-96f4-5894064fcd5f","Type":"ContainerStarted","Data":"954e4a51ea3d62f86f399e8940758c38ab1d002739fe50b1a825721f28ff3407"}
Sep 15 09:31:24.525250 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:24.525207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw" event={"ID":"402cfbbd-e89a-41c4-96f4-5894064fcd5f","Type":"ContainerStarted","Data":"e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea"}
Sep 15 09:31:24.525694 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:24.525376    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:24.542721 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:24.542647    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw" podStartSLOduration=2.43160024 podStartE2EDuration="3.542628604s" totalImagesPullingTime="1.111028364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:31:24.542301112 +0000 UTC m=+1048.372093017" watchObservedRunningTime="2026-09-15 09:31:24.542628604 +0000 UTC m=+1048.372420507"
Sep 15 09:31:29.418416 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:29.418380    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:31:30.535563 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:30.535533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:31:45.590211 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.590170    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"]
Sep 15 09:31:45.594019 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.593992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.596384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.596360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c183a5\"/\"premium-w4-c183a5-kserve-self-signed-certs\""
Sep 15 09:31:45.603107 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.603072    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"]
Sep 15 09:31:45.661547 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.661504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b750eca-e525-4275-8978-8c459e9cef2b-tls-certs\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.661739 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.661627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-model-cache\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.661739 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.661681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-dshm\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.661739 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.661702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-home\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.661739 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.661723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-tmp-dir\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.661900 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.661763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdsc\" (UniqueName: \"kubernetes.io/projected/7b750eca-e525-4275-8978-8c459e9cef2b-kube-api-access-4pdsc\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.762541 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.762492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-model-cache\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.762736 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.762676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-dshm\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.762736 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.762723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-home\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.762861 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.762752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-tmp-dir\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.762861 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.762813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdsc\" (UniqueName: \"kubernetes.io/projected/7b750eca-e525-4275-8978-8c459e9cef2b-kube-api-access-4pdsc\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.763017 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.762993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-model-cache\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.763091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.763050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b750eca-e525-4275-8978-8c459e9cef2b-tls-certs\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.763162 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.763136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-home\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.763553 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.763524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-tmp-dir\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.765648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.765623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-dshm\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.765832 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.765815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b750eca-e525-4275-8978-8c459e9cef2b-tls-certs\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.771522 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.771493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdsc\" (UniqueName: \"kubernetes.io/projected/7b750eca-e525-4275-8978-8c459e9cef2b-kube-api-access-4pdsc\") pod \"premium-w4-c183a5-kserve-6c7c99dc7-2q2c9\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") " pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:45.904661 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:45.904557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:46.180011 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:46.179978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"]
Sep 15 09:31:46.182751 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:31:46.182713    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b750eca_e525_4275_8978_8c459e9cef2b.slice/crio-b092c3a368e704e2835d3950aa27ede21d4ecafeb12ae9dcedba99bfb2297ede WatchSource:0}: Error finding container b092c3a368e704e2835d3950aa27ede21d4ecafeb12ae9dcedba99bfb2297ede: Status 404 returned error can't find the container with id b092c3a368e704e2835d3950aa27ede21d4ecafeb12ae9dcedba99bfb2297ede
Sep 15 09:31:46.637453 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:46.637361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9" event={"ID":"7b750eca-e525-4275-8978-8c459e9cef2b","Type":"ContainerStarted","Data":"b31b9148567de1664f78afea9c3683b76ba0dbae2637b71c5c36d06ab66ea101"}
Sep 15 09:31:46.637453 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:46.637407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9" event={"ID":"7b750eca-e525-4275-8978-8c459e9cef2b","Type":"ContainerStarted","Data":"b092c3a368e704e2835d3950aa27ede21d4ecafeb12ae9dcedba99bfb2297ede"}
Sep 15 09:31:46.655673 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:46.655614    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9" podStartSLOduration=1.655592468 podStartE2EDuration="1.655592468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:31:46.653030561 +0000 UTC m=+1070.482822463" watchObservedRunningTime="2026-09-15 09:31:46.655592468 +0000 UTC m=+1070.485384372"
Sep 15 09:31:52.298270 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:52.298191    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:31:55.905463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:55.905411    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:55.905886 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:55.905479    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:55.918481 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:55.918453    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:31:56.709099 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:31:56.709071    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:32:09.995514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:09.995474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:32:27.308184 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.308150    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"]
Sep 15 09:32:27.311777 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.311752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.316373 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.316352    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8df9f24a-serving-cert\""
Sep 15 09:32:27.332983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.332958    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"]
Sep 15 09:32:27.345574 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.345549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-maas-api-tls\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.345667 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.345604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-metrics-certs\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.345667 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.345647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f3c4f7-cc84-4611-8df5-e0837e5e714d-openshift-service-ca\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.345743 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.345681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vbq7\" (UniqueName: \"kubernetes.io/projected/82f3c4f7-cc84-4611-8df5-e0837e5e714d-kube-api-access-4vbq7\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.446964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.446935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-maas-api-tls\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.447086 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.446970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-metrics-certs\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.447086 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.446994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f3c4f7-cc84-4611-8df5-e0837e5e714d-openshift-service-ca\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.447086 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.447020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vbq7\" (UniqueName: \"kubernetes.io/projected/82f3c4f7-cc84-4611-8df5-e0837e5e714d-kube-api-access-4vbq7\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.447086 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:32:27.447078    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-serving-cert: secret "maas-api-e2e-mt-8df9f24a-serving-cert" not found
Sep 15 09:32:27.447262 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:32:27.447152    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-maas-api-tls podName:82f3c4f7-cc84-4611-8df5-e0837e5e714d nodeName:}" failed. No retries permitted until 2026-09-15 09:32:27.947137361 +0000 UTC m=+1111.776929255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-maas-api-tls") pod "maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l" (UID: "82f3c4f7-cc84-4611-8df5-e0837e5e714d") : secret "maas-api-e2e-mt-8df9f24a-serving-cert" not found
Sep 15 09:32:27.447589 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.447567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f3c4f7-cc84-4611-8df5-e0837e5e714d-openshift-service-ca\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.449436 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.449413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-metrics-certs\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.459647 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.459630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vbq7\" (UniqueName: \"kubernetes.io/projected/82f3c4f7-cc84-4611-8df5-e0837e5e714d-kube-api-access-4vbq7\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.951377 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.951321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-maas-api-tls\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:27.953786 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:27.953754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-maas-api-tls\") pod \"maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:28.221894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:28.221805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:28.492022 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:28.491996    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"]
Sep 15 09:32:28.494076 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:32:28.494050    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f3c4f7_cc84_4611_8df5_e0837e5e714d.slice/crio-6f83532df2ca7c84312e314d0c0c90590eaf15d82712e6c13ca2473685c4bfae WatchSource:0}: Error finding container 6f83532df2ca7c84312e314d0c0c90590eaf15d82712e6c13ca2473685c4bfae: Status 404 returned error can't find the container with id 6f83532df2ca7c84312e314d0c0c90590eaf15d82712e6c13ca2473685c4bfae
Sep 15 09:32:28.863502 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:28.863417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l" event={"ID":"82f3c4f7-cc84-4611-8df5-e0837e5e714d","Type":"ContainerStarted","Data":"6f83532df2ca7c84312e314d0c0c90590eaf15d82712e6c13ca2473685c4bfae"}
Sep 15 09:32:29.870100 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:29.870056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l" event={"ID":"82f3c4f7-cc84-4611-8df5-e0837e5e714d","Type":"ContainerStarted","Data":"8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d"}
Sep 15 09:32:29.870581 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:29.870129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:29.888318 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:29.888268    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l" podStartSLOduration=2.412716633 podStartE2EDuration="2.888252763s" totalImagesPullingTime="475.53613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:32:29.885579758 +0000 UTC m=+1113.715371661" watchObservedRunningTime="2026-09-15 09:32:29.888252763 +0000 UTC m=+1113.718044666"
Sep 15 09:32:35.880658 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:35.880620    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:32:36.200634 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.200587    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"]
Sep 15 09:32:36.205265 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.205237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.207497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.207464    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0e280a\"/\"default-dockercfg-zjl2f\""
Sep 15 09:32:36.207497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.207497    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0e280a\"/\"kube-root-ca.crt\""
Sep 15 09:32:36.208633 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.208603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0e280a\"/\"openshift-service-ca.crt\""
Sep 15 09:32:36.208838 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.208816    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0e280a\"/\"test-model-0e280a-kserve-self-signed-certs\""
Sep 15 09:32:36.211446 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.211420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"]
Sep 15 09:32:36.335145 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.335095    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r2g\" (UniqueName: \"kubernetes.io/projected/6920dbda-ab79-4102-b999-040ce31c9e9e-kube-api-access-64r2g\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.335389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.335156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-dshm\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.335389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.335251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-model-cache\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.335389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.335285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6920dbda-ab79-4102-b999-040ce31c9e9e-tls-certs\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.335533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.335436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-tmp-dir\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.335533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.335480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-home\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.436494 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.436442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-model-cache\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.436494 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.436491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6920dbda-ab79-4102-b999-040ce31c9e9e-tls-certs\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.436780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.436555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-tmp-dir\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.436780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.436594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-home\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.436780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.436649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r2g\" (UniqueName: \"kubernetes.io/projected/6920dbda-ab79-4102-b999-040ce31c9e9e-kube-api-access-64r2g\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.436780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.436684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-dshm\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.436993 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.436896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-model-cache\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.436993 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.436965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-tmp-dir\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.437096 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.436989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-home\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.439088 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.439059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-dshm\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.439393 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.439368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6920dbda-ab79-4102-b999-040ce31c9e9e-tls-certs\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.444940 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.444899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r2g\" (UniqueName: \"kubernetes.io/projected/6920dbda-ab79-4102-b999-040ce31c9e9e-kube-api-access-64r2g\") pod \"test-model-0e280a-kserve-775594b5c7-h78ll\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") " pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.518742 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.518632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:36.793985 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.793959    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"]
Sep 15 09:32:36.796317 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:32:36.796290    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6920dbda_ab79_4102_b999_040ce31c9e9e.slice/crio-216209ca1a07865bbe9620e5ed40b28a28d5af42fea52a6eddb0c4ca9c15dd79 WatchSource:0}: Error finding container 216209ca1a07865bbe9620e5ed40b28a28d5af42fea52a6eddb0c4ca9c15dd79: Status 404 returned error can't find the container with id 216209ca1a07865bbe9620e5ed40b28a28d5af42fea52a6eddb0c4ca9c15dd79
Sep 15 09:32:36.911832 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.911790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll" event={"ID":"6920dbda-ab79-4102-b999-040ce31c9e9e","Type":"ContainerStarted","Data":"15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140"}
Sep 15 09:32:36.912324 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.911847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll" event={"ID":"6920dbda-ab79-4102-b999-040ce31c9e9e","Type":"ContainerStarted","Data":"216209ca1a07865bbe9620e5ed40b28a28d5af42fea52a6eddb0c4ca9c15dd79"}
Sep 15 09:32:36.927657 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:36.927606    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll" podStartSLOduration=0.927588235 podStartE2EDuration="927.588235ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:32:36.926034774 +0000 UTC m=+1120.755826678" watchObservedRunningTime="2026-09-15 09:32:36.927588235 +0000 UTC m=+1120.757380138"
Sep 15 09:32:37.794961 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:37.794922    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:32:46.519399 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:46.519364    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:46.519802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:46.519410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:46.531864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:46.531841    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:32:46.974197 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:32:46.974168    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:33:01.107534 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.107497    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"]
Sep 15 09:33:01.115958 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.115395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.120000 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.119082    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5b654a\"/\"default-dockercfg-kjj4h\""
Sep 15 09:33:01.120560 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.120532    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5b654a\"/\"kube-root-ca.crt\""
Sep 15 09:33:01.120727 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.120572    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5b654a\"/\"explicit-model-5b654a-kserve-self-signed-certs\""
Sep 15 09:33:01.121071 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.121044    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"]
Sep 15 09:33:01.121466 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.121439    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5b654a\"/\"openshift-service-ca.crt\""
Sep 15 09:33:01.167820 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.167785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-model-cache\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.167820 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.167830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-dshm\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.168050 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.167936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts69n\" (UniqueName: \"kubernetes.io/projected/5e3d853c-6b5f-4930-a49d-85840d5714d1-kube-api-access-ts69n\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.168050 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.167972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3d853c-6b5f-4930-a49d-85840d5714d1-tls-certs\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.168050 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.168040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-home\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.168161 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.168056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-tmp-dir\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.269131 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.269083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-dshm\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.269369 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.269154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts69n\" (UniqueName: \"kubernetes.io/projected/5e3d853c-6b5f-4930-a49d-85840d5714d1-kube-api-access-ts69n\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.269369 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.269173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3d853c-6b5f-4930-a49d-85840d5714d1-tls-certs\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.269369 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.269236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-home\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.269369 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.269251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-tmp-dir\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.269369 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.269298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-model-cache\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.269688 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.269637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-home\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.269688 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.269645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-model-cache\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.269765 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.269685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-tmp-dir\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.271498 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.271480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-dshm\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.271880 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.271858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3d853c-6b5f-4930-a49d-85840d5714d1-tls-certs\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.276735 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.276713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts69n\" (UniqueName: \"kubernetes.io/projected/5e3d853c-6b5f-4930-a49d-85840d5714d1-kube-api-access-ts69n\") pod \"explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") " pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.428781 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.428744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:01.701422 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:01.701353    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"]
Sep 15 09:33:01.703830 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:33:01.703792    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e3d853c_6b5f_4930_a49d_85840d5714d1.slice/crio-db06fcfec8fa7cb5fe2336857f798392217278bff2d794ca56c02220f99f4314 WatchSource:0}: Error finding container db06fcfec8fa7cb5fe2336857f798392217278bff2d794ca56c02220f99f4314: Status 404 returned error can't find the container with id db06fcfec8fa7cb5fe2336857f798392217278bff2d794ca56c02220f99f4314
Sep 15 09:33:02.041104 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:02.041011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl" event={"ID":"5e3d853c-6b5f-4930-a49d-85840d5714d1","Type":"ContainerStarted","Data":"76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022"}
Sep 15 09:33:02.041104 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:02.041058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl" event={"ID":"5e3d853c-6b5f-4930-a49d-85840d5714d1","Type":"ContainerStarted","Data":"db06fcfec8fa7cb5fe2336857f798392217278bff2d794ca56c02220f99f4314"}
Sep 15 09:33:02.061570 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:02.061511    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl" podStartSLOduration=1.061492497 podStartE2EDuration="1.061492497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:33:02.058820857 +0000 UTC m=+1145.888612760" watchObservedRunningTime="2026-09-15 09:33:02.061492497 +0000 UTC m=+1145.891284443"
Sep 15 09:33:02.498881 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:02.498840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:33:10.985935 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:10.985900    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"]
Sep 15 09:33:10.986450 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:10.986169    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l" podUID="82f3c4f7-cc84-4611-8df5-e0837e5e714d" containerName="maas-api" containerID="cri-o://8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d" gracePeriod=30
Sep 15 09:33:11.235872 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.235841    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:33:11.366956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.366874    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vbq7\" (UniqueName: \"kubernetes.io/projected/82f3c4f7-cc84-4611-8df5-e0837e5e714d-kube-api-access-4vbq7\") pod \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") "
Sep 15 09:33:11.367112 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.366968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-metrics-certs\") pod \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") "
Sep 15 09:33:11.367112 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.367026    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-maas-api-tls\") pod \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") "
Sep 15 09:33:11.367112 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.367108    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f3c4f7-cc84-4611-8df5-e0837e5e714d-openshift-service-ca\") pod \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\" (UID: \"82f3c4f7-cc84-4611-8df5-e0837e5e714d\") "
Sep 15 09:33:11.367761 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.367720    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f3c4f7-cc84-4611-8df5-e0837e5e714d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82f3c4f7-cc84-4611-8df5-e0837e5e714d" (UID: "82f3c4f7-cc84-4611-8df5-e0837e5e714d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:33:11.369274 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.369233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "82f3c4f7-cc84-4611-8df5-e0837e5e714d" (UID: "82f3c4f7-cc84-4611-8df5-e0837e5e714d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:11.369599 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.369562    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82f3c4f7-cc84-4611-8df5-e0837e5e714d" (UID: "82f3c4f7-cc84-4611-8df5-e0837e5e714d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:11.369599 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.369573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f3c4f7-cc84-4611-8df5-e0837e5e714d-kube-api-access-4vbq7" (OuterVolumeSpecName: "kube-api-access-4vbq7") pod "82f3c4f7-cc84-4611-8df5-e0837e5e714d" (UID: "82f3c4f7-cc84-4611-8df5-e0837e5e714d"). InnerVolumeSpecName "kube-api-access-4vbq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:11.429513 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.429491    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:11.429626 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.429526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:11.441710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.441688    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:11.468829 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.468806    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vbq7\" (UniqueName: \"kubernetes.io/projected/82f3c4f7-cc84-4611-8df5-e0837e5e714d-kube-api-access-4vbq7\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:33:11.468829 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.468829    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:33:11.468957 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.468839    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f3c4f7-cc84-4611-8df5-e0837e5e714d-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:33:11.468957 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:11.468847    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f3c4f7-cc84-4611-8df5-e0837e5e714d-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:33:12.097947 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.097912    2579 generic.go:356] "Generic (PLEG): container finished" podID="82f3c4f7-cc84-4611-8df5-e0837e5e714d" containerID="8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d" exitCode=0
Sep 15 09:33:12.098439 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.097971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"
Sep 15 09:33:12.098439 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.097992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l" event={"ID":"82f3c4f7-cc84-4611-8df5-e0837e5e714d","Type":"ContainerDied","Data":"8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d"}
Sep 15 09:33:12.098439 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.098054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l" event={"ID":"82f3c4f7-cc84-4611-8df5-e0837e5e714d","Type":"ContainerDied","Data":"6f83532df2ca7c84312e314d0c0c90590eaf15d82712e6c13ca2473685c4bfae"}
Sep 15 09:33:12.098439 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.098088    2579 scope.go:117] "RemoveContainer" containerID="8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d"
Sep 15 09:33:12.107820 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.107802    2579 scope.go:117] "RemoveContainer" containerID="8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d"
Sep 15 09:33:12.108097 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:33:12.108079    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d\": container with ID starting with 8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d not found: ID does not exist" containerID="8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d"
Sep 15 09:33:12.108148 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.108105    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d"} err="failed to get container status \"8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d\": rpc error: code = NotFound desc = could not find container \"8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d\": container with ID starting with 8dff0f8b523640d55d445575807d45a87dda485417ebebfc43c6ed2a11da5d0d not found: ID does not exist"
Sep 15 09:33:12.110508 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.110489    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:33:12.118301 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.118281    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"]
Sep 15 09:33:12.121514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.121495    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8df9f24a-7bc9d9d797-ltv2l"]
Sep 15 09:33:12.678121 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:12.678089    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f3c4f7-cc84-4611-8df5-e0837e5e714d" path="/var/lib/kubelet/pods/82f3c4f7-cc84-4611-8df5-e0837e5e714d/volumes"
Sep 15 09:33:21.108785 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:21.108704    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:33:37.708910 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.708879    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"]
Sep 15 09:33:37.709513 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.709487    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f3c4f7-cc84-4611-8df5-e0837e5e714d" containerName="maas-api"
Sep 15 09:33:37.709688 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.709675    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f3c4f7-cc84-4611-8df5-e0837e5e714d" containerName="maas-api"
Sep 15 09:33:37.709941 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.709890    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f3c4f7-cc84-4611-8df5-e0837e5e714d" containerName="maas-api"
Sep 15 09:33:37.713228 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.713207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.715128 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.715109    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f514a4-serving-cert\""
Sep 15 09:33:37.723660 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.723634    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"]
Sep 15 09:33:37.810660 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.810633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.810806 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.810709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431c1687-f5e9-4a67-9d00-918ed8813622-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.810806 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.810739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtlp7\" (UniqueName: \"kubernetes.io/projected/431c1687-f5e9-4a67-9d00-918ed8813622-kube-api-access-wtlp7\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.810882 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.810816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.912233 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.912189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.912400 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.912264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.912400 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.912373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431c1687-f5e9-4a67-9d00-918ed8813622-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.912509 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.912410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlp7\" (UniqueName: \"kubernetes.io/projected/431c1687-f5e9-4a67-9d00-918ed8813622-kube-api-access-wtlp7\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.913093 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.913071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431c1687-f5e9-4a67-9d00-918ed8813622-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.914752 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.914721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.914837 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.914764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:37.919496 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:37.919476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtlp7\" (UniqueName: \"kubernetes.io/projected/431c1687-f5e9-4a67-9d00-918ed8813622-kube-api-access-wtlp7\") pod \"maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:38.025063 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:38.024990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:38.303486 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:33:38.303451    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431c1687_f5e9_4a67_9d00_918ed8813622.slice/crio-88329e11f9f5fe52f70c7a046496f3aecd1a571d47e3ee49e7938f44c45bed4f WatchSource:0}: Error finding container 88329e11f9f5fe52f70c7a046496f3aecd1a571d47e3ee49e7938f44c45bed4f: Status 404 returned error can't find the container with id 88329e11f9f5fe52f70c7a046496f3aecd1a571d47e3ee49e7938f44c45bed4f
Sep 15 09:33:38.310847 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:38.310819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"]
Sep 15 09:33:39.234766 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:39.234726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9" event={"ID":"431c1687-f5e9-4a67-9d00-918ed8813622","Type":"ContainerStarted","Data":"2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424"}
Sep 15 09:33:39.235151 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:39.234786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9" event={"ID":"431c1687-f5e9-4a67-9d00-918ed8813622","Type":"ContainerStarted","Data":"88329e11f9f5fe52f70c7a046496f3aecd1a571d47e3ee49e7938f44c45bed4f"}
Sep 15 09:33:39.235151 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:39.234846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:39.251872 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:39.251811    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9" podStartSLOduration=1.792631293 podStartE2EDuration="2.251794991s" totalImagesPullingTime="459.163698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:33:39.250394619 +0000 UTC m=+1183.080186523" watchObservedRunningTime="2026-09-15 09:33:39.251794991 +0000 UTC m=+1183.081586893"
Sep 15 09:33:45.245380 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:45.245351    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:33:45.794818 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:45.794786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:33:47.305540 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.305506    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"]
Sep 15 09:33:47.306002 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.305759    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw" podUID="402cfbbd-e89a-41c4-96f4-5894064fcd5f" containerName="maas-api" containerID="cri-o://e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea" gracePeriod=30
Sep 15 09:33:47.555161 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.555129    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:33:47.600805 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.600723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-metrics-certs\") pod \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") "
Sep 15 09:33:47.600805 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.600785    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9nfv\" (UniqueName: \"kubernetes.io/projected/402cfbbd-e89a-41c4-96f4-5894064fcd5f-kube-api-access-w9nfv\") pod \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") "
Sep 15 09:33:47.601022 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.600861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402cfbbd-e89a-41c4-96f4-5894064fcd5f-openshift-service-ca\") pod \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") "
Sep 15 09:33:47.601022 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.600897    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-maas-api-tls\") pod \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\" (UID: \"402cfbbd-e89a-41c4-96f4-5894064fcd5f\") "
Sep 15 09:33:47.601274 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.601238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402cfbbd-e89a-41c4-96f4-5894064fcd5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "402cfbbd-e89a-41c4-96f4-5894064fcd5f" (UID: "402cfbbd-e89a-41c4-96f4-5894064fcd5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:33:47.603026 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.602992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "402cfbbd-e89a-41c4-96f4-5894064fcd5f" (UID: "402cfbbd-e89a-41c4-96f4-5894064fcd5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:47.603390 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.603364    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "402cfbbd-e89a-41c4-96f4-5894064fcd5f" (UID: "402cfbbd-e89a-41c4-96f4-5894064fcd5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:47.603557 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.603532    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402cfbbd-e89a-41c4-96f4-5894064fcd5f-kube-api-access-w9nfv" (OuterVolumeSpecName: "kube-api-access-w9nfv") pod "402cfbbd-e89a-41c4-96f4-5894064fcd5f" (UID: "402cfbbd-e89a-41c4-96f4-5894064fcd5f"). InnerVolumeSpecName "kube-api-access-w9nfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:47.702234 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.702209    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402cfbbd-e89a-41c4-96f4-5894064fcd5f-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:33:47.702234 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.702233    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:33:47.702405 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.702243    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402cfbbd-e89a-41c4-96f4-5894064fcd5f-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:33:47.702405 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:47.702251    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9nfv\" (UniqueName: \"kubernetes.io/projected/402cfbbd-e89a-41c4-96f4-5894064fcd5f-kube-api-access-w9nfv\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:33:48.286074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.286035    2579 generic.go:356] "Generic (PLEG): container finished" podID="402cfbbd-e89a-41c4-96f4-5894064fcd5f" containerID="e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea" exitCode=0
Sep 15 09:33:48.286259 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.286085    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"
Sep 15 09:33:48.286259 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.286090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw" event={"ID":"402cfbbd-e89a-41c4-96f4-5894064fcd5f","Type":"ContainerDied","Data":"e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea"}
Sep 15 09:33:48.286259 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.286153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw" event={"ID":"402cfbbd-e89a-41c4-96f4-5894064fcd5f","Type":"ContainerDied","Data":"954e4a51ea3d62f86f399e8940758c38ab1d002739fe50b1a825721f28ff3407"}
Sep 15 09:33:48.286259 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.286186    2579 scope.go:117] "RemoveContainer" containerID="e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea"
Sep 15 09:33:48.295368 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.295332    2579 scope.go:117] "RemoveContainer" containerID="e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea"
Sep 15 09:33:48.295629 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:33:48.295613    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea\": container with ID starting with e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea not found: ID does not exist" containerID="e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea"
Sep 15 09:33:48.295673 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.295637    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea"} err="failed to get container status \"e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea\": rpc error: code = NotFound desc = could not find container \"e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea\": container with ID starting with e7d0e427dd5a4dd22cf397ed4de6199d300d879afeaa8190c777f36946fc6eea not found: ID does not exist"
Sep 15 09:33:48.306231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.306208    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"]
Sep 15 09:33:48.311864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.311844    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b654a-6d576cbcf7-4f2fw"]
Sep 15 09:33:48.676503 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:48.676475    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="402cfbbd-e89a-41c4-96f4-5894064fcd5f" path="/var/lib/kubelet/pods/402cfbbd-e89a-41c4-96f4-5894064fcd5f/volumes"
Sep 15 09:33:52.870415 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:33:52.870381    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"]
Sep 15 09:34:02.622705 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.622668    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"]
Sep 15 09:34:02.623158 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.623078    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402cfbbd-e89a-41c4-96f4-5894064fcd5f" containerName="maas-api"
Sep 15 09:34:02.623158 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.623089    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="402cfbbd-e89a-41c4-96f4-5894064fcd5f" containerName="maas-api"
Sep 15 09:34:02.623275 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.623163    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="402cfbbd-e89a-41c4-96f4-5894064fcd5f" containerName="maas-api"
Sep 15 09:34:02.627967 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.627943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.630083 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.630059    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-161187-serving-cert\""
Sep 15 09:34:02.634767 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.634744    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"]
Sep 15 09:34:02.741925 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.741898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.742069 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.741934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzrg\" (UniqueName: \"kubernetes.io/projected/f31ec4b0-5806-4388-a725-8cccef3f2f85-kube-api-access-zqzrg\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.742069 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.741984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.742146 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.742107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ec4b0-5806-4388-a725-8cccef3f2f85-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.842568 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.842537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ec4b0-5806-4388-a725-8cccef3f2f85-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.842704 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.842605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.842704 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.842626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzrg\" (UniqueName: \"kubernetes.io/projected/f31ec4b0-5806-4388-a725-8cccef3f2f85-kube-api-access-zqzrg\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.842704 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.842653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.842879 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:34:02.842754    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-serving-cert: secret "maas-api-e2e-shared-b-w2-161187-serving-cert" not found
Sep 15 09:34:02.842879 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:34:02.842810    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-maas-api-tls podName:f31ec4b0-5806-4388-a725-8cccef3f2f85 nodeName:}" failed. No retries permitted until 2026-09-15 09:34:03.342795513 +0000 UTC m=+1207.172587394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-maas-api-tls") pod "maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt" (UID: "f31ec4b0-5806-4388-a725-8cccef3f2f85") : secret "maas-api-e2e-shared-b-w2-161187-serving-cert" not found
Sep 15 09:34:02.843266 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.843242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ec4b0-5806-4388-a725-8cccef3f2f85-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.845045 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.845027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:02.861262 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:02.861234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzrg\" (UniqueName: \"kubernetes.io/projected/f31ec4b0-5806-4388-a725-8cccef3f2f85-kube-api-access-zqzrg\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:03.347920 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:03.347875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:03.350172 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:03.350145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:03.542149 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:03.542122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:03.812658 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:03.812628    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"]
Sep 15 09:34:03.814416 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:34:03.814388    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31ec4b0_5806_4388_a725_8cccef3f2f85.slice/crio-4538e45e05b220f5d9e4fdde22e6e69951a06e0118de711caa49d887c5e2affc WatchSource:0}: Error finding container 4538e45e05b220f5d9e4fdde22e6e69951a06e0118de711caa49d887c5e2affc: Status 404 returned error can't find the container with id 4538e45e05b220f5d9e4fdde22e6e69951a06e0118de711caa49d887c5e2affc
Sep 15 09:34:04.380167 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:04.380126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt" event={"ID":"f31ec4b0-5806-4388-a725-8cccef3f2f85","Type":"ContainerStarted","Data":"6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637"}
Sep 15 09:34:04.380305 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:04.380191    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:04.380305 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:04.380214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt" event={"ID":"f31ec4b0-5806-4388-a725-8cccef3f2f85","Type":"ContainerStarted","Data":"4538e45e05b220f5d9e4fdde22e6e69951a06e0118de711caa49d887c5e2affc"}
Sep 15 09:34:04.397106 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:04.397059    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt" podStartSLOduration=1.935030085 podStartE2EDuration="2.397044543s" totalImagesPullingTime="462.014458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:34:04.395383362 +0000 UTC m=+1208.225175265" watchObservedRunningTime="2026-09-15 09:34:04.397044543 +0000 UTC m=+1208.226836445"
Sep 15 09:34:07.890108 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:07.890066    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl" podUID="5e3d853c-6b5f-4930-a49d-85840d5714d1" containerName="main" containerID="cri-o://76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022" gracePeriod=45
Sep 15 09:34:08.142631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.142577    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:34:08.295913 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.295878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3d853c-6b5f-4930-a49d-85840d5714d1-tls-certs\") pod \"5e3d853c-6b5f-4930-a49d-85840d5714d1\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") "
Sep 15 09:34:08.296089 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.295922    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts69n\" (UniqueName: \"kubernetes.io/projected/5e3d853c-6b5f-4930-a49d-85840d5714d1-kube-api-access-ts69n\") pod \"5e3d853c-6b5f-4930-a49d-85840d5714d1\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") "
Sep 15 09:34:08.296089 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.295953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-dshm\") pod \"5e3d853c-6b5f-4930-a49d-85840d5714d1\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") "
Sep 15 09:34:08.296089 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.296066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-model-cache\") pod \"5e3d853c-6b5f-4930-a49d-85840d5714d1\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") "
Sep 15 09:34:08.296262 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.296157    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-tmp-dir\") pod \"5e3d853c-6b5f-4930-a49d-85840d5714d1\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") "
Sep 15 09:34:08.296262 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.296206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-home\") pod \"5e3d853c-6b5f-4930-a49d-85840d5714d1\" (UID: \"5e3d853c-6b5f-4930-a49d-85840d5714d1\") "
Sep 15 09:34:08.296413 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.296355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e3d853c-6b5f-4930-a49d-85840d5714d1" (UID: "5e3d853c-6b5f-4930-a49d-85840d5714d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:34:08.296502 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.296427    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e3d853c-6b5f-4930-a49d-85840d5714d1" (UID: "5e3d853c-6b5f-4930-a49d-85840d5714d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:34:08.296549 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.296504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-home" (OuterVolumeSpecName: "home") pod "5e3d853c-6b5f-4930-a49d-85840d5714d1" (UID: "5e3d853c-6b5f-4930-a49d-85840d5714d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:34:08.296701 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.296681    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:34:08.296785 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.296701    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:34:08.296785 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.296710    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:34:08.298088 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.298069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-dshm" (OuterVolumeSpecName: "dshm") pod "5e3d853c-6b5f-4930-a49d-85840d5714d1" (UID: "5e3d853c-6b5f-4930-a49d-85840d5714d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:34:08.298197 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.298181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3d853c-6b5f-4930-a49d-85840d5714d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e3d853c-6b5f-4930-a49d-85840d5714d1" (UID: "5e3d853c-6b5f-4930-a49d-85840d5714d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:34:08.298242 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.298224    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3d853c-6b5f-4930-a49d-85840d5714d1-kube-api-access-ts69n" (OuterVolumeSpecName: "kube-api-access-ts69n") pod "5e3d853c-6b5f-4930-a49d-85840d5714d1" (UID: "5e3d853c-6b5f-4930-a49d-85840d5714d1"). InnerVolumeSpecName "kube-api-access-ts69n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:34:08.397775 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.397714    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3d853c-6b5f-4930-a49d-85840d5714d1-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:34:08.397775 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.397747    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts69n\" (UniqueName: \"kubernetes.io/projected/5e3d853c-6b5f-4930-a49d-85840d5714d1-kube-api-access-ts69n\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:34:08.397775 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.397760    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3d853c-6b5f-4930-a49d-85840d5714d1-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:34:08.401967 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.401942    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e3d853c-6b5f-4930-a49d-85840d5714d1" containerID="76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022" exitCode=0
Sep 15 09:34:08.402093 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.401991    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"
Sep 15 09:34:08.402093 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.402014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl" event={"ID":"5e3d853c-6b5f-4930-a49d-85840d5714d1","Type":"ContainerDied","Data":"76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022"}
Sep 15 09:34:08.402093 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.402045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl" event={"ID":"5e3d853c-6b5f-4930-a49d-85840d5714d1","Type":"ContainerDied","Data":"db06fcfec8fa7cb5fe2336857f798392217278bff2d794ca56c02220f99f4314"}
Sep 15 09:34:08.402093 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.402061    2579 scope.go:117] "RemoveContainer" containerID="76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022"
Sep 15 09:34:08.413951 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.413933    2579 scope.go:117] "RemoveContainer" containerID="76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022"
Sep 15 09:34:08.414275 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:34:08.414250    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022\": container with ID starting with 76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022 not found: ID does not exist" containerID="76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022"
Sep 15 09:34:08.414329 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.414284    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022"} err="failed to get container status \"76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022\": rpc error: code = NotFound desc = could not find container \"76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022\": container with ID starting with 76a3174e1d39ab6bf1124705af1911113b5790c0d30e07c01f649a8bdfbcc022 not found: ID does not exist"
Sep 15 09:34:08.424107 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.424083    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"]
Sep 15 09:34:08.427536 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.427514    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-5b654a/explicit-model-5b654a-kserve-5d7bbc6bf9-6fxkl"]
Sep 15 09:34:08.677196 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:08.677168    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e3d853c-6b5f-4930-a49d-85840d5714d1" path="/var/lib/kubelet/pods/5e3d853c-6b5f-4930-a49d-85840d5714d1/volumes"
Sep 15 09:34:10.390693 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:10.390656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:34:19.096399 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:19.096363    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:34:33.584175 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.584145    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"]
Sep 15 09:34:33.584636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.584620    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3d853c-6b5f-4930-a49d-85840d5714d1" containerName="main"
Sep 15 09:34:33.584636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.584637    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3d853c-6b5f-4930-a49d-85840d5714d1" containerName="main"
Sep 15 09:34:33.584776 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.584734    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3d853c-6b5f-4930-a49d-85840d5714d1" containerName="main"
Sep 15 09:34:33.587944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.587928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.590074 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.590053    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c183a5\"/\"e2e-conflict-a-0b195997-kserve-self-signed-certs\""
Sep 15 09:34:33.601208 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.599732    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"]
Sep 15 09:34:33.630630 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.630593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-dshm\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.630726 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.630645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgdqq\" (UniqueName: \"kubernetes.io/projected/80f35a0d-166c-4577-b169-d811e61448ad-kube-api-access-sgdqq\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.630769 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.630739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35a0d-166c-4577-b169-d811e61448ad-tls-certs\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.630895 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.630874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-tmp-dir\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.630997 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.630978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-home\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.631067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.631048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-model-cache\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.732463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.732433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-model-cache\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.732597 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.732490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-dshm\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.732663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.732597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgdqq\" (UniqueName: \"kubernetes.io/projected/80f35a0d-166c-4577-b169-d811e61448ad-kube-api-access-sgdqq\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.732663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.732630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35a0d-166c-4577-b169-d811e61448ad-tls-certs\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.732774 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.732698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-tmp-dir\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.732774 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.732754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-home\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.732952 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.732925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-model-cache\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.733135 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.733106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-tmp-dir\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.733247 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.733204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-home\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.735017 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.734998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-dshm\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.735113 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.735019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35a0d-166c-4577-b169-d811e61448ad-tls-certs\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.739751 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.739727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgdqq\" (UniqueName: \"kubernetes.io/projected/80f35a0d-166c-4577-b169-d811e61448ad-kube-api-access-sgdqq\") pod \"e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") " pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:33.904362 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:33.904272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:34.174739 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:34.174715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"]
Sep 15 09:34:34.177256 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:34:34.177231    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80f35a0d_166c_4577_b169_d811e61448ad.slice/crio-9ee02ec044b5a66acf75bd925afe4faae1105058890ca5e55894658f81b31d7a WatchSource:0}: Error finding container 9ee02ec044b5a66acf75bd925afe4faae1105058890ca5e55894658f81b31d7a: Status 404 returned error can't find the container with id 9ee02ec044b5a66acf75bd925afe4faae1105058890ca5e55894658f81b31d7a
Sep 15 09:34:34.536237 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:34.536154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb" event={"ID":"80f35a0d-166c-4577-b169-d811e61448ad","Type":"ContainerStarted","Data":"ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e"}
Sep 15 09:34:34.536237 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:34.536196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb" event={"ID":"80f35a0d-166c-4577-b169-d811e61448ad","Type":"ContainerStarted","Data":"9ee02ec044b5a66acf75bd925afe4faae1105058890ca5e55894658f81b31d7a"}
Sep 15 09:34:34.553780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:34.553732    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb" podStartSLOduration=1.553717823 podStartE2EDuration="1.553717823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:34:34.552844053 +0000 UTC m=+1238.382635955" watchObservedRunningTime="2026-09-15 09:34:34.553717823 +0000 UTC m=+1238.383509726"
Sep 15 09:34:43.904622 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:43.904590    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:43.905083 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:43.904832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:43.917588 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:43.917568    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:44.600746 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:44.600719    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:34:47.438262 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.438228    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"]
Sep 15 09:34:47.442671 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.442646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.444798 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.444776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-312334ca-serving-cert\""
Sep 15 09:34:47.452646 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.452621    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"]
Sep 15 09:34:47.569603 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.569570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4czp\" (UniqueName: \"kubernetes.io/projected/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-kube-api-access-f4czp\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.569785 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.569722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-metrics-certs\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.569785 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.569776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-maas-api-tls\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.569908 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.569857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-openshift-service-ca\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.670654 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.670622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-metrics-certs\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.670799 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.670662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-maas-api-tls\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.670799 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.670698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-openshift-service-ca\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.670799 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.670723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4czp\" (UniqueName: \"kubernetes.io/projected/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-kube-api-access-f4czp\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.671328 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.671309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-openshift-service-ca\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.673015 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.672990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-maas-api-tls\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.673182 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.673164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-metrics-certs\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.678852 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.678832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4czp\" (UniqueName: \"kubernetes.io/projected/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-kube-api-access-f4czp\") pod \"maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:47.759251 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:47.759193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:48.041120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:48.041086    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"]
Sep 15 09:34:48.042791 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:34:48.042735    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbce62e5_6932_4ab5_b5aa_3525dc3a2398.slice/crio-0af4d355c9be1895938ddd9c0204f91435111aba63f2f4e69a17112f03300056 WatchSource:0}: Error finding container 0af4d355c9be1895938ddd9c0204f91435111aba63f2f4e69a17112f03300056: Status 404 returned error can't find the container with id 0af4d355c9be1895938ddd9c0204f91435111aba63f2f4e69a17112f03300056
Sep 15 09:34:48.611187 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:48.611153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl" event={"ID":"dbce62e5-6932-4ab5-b5aa-3525dc3a2398","Type":"ContainerStarted","Data":"0af4d355c9be1895938ddd9c0204f91435111aba63f2f4e69a17112f03300056"}
Sep 15 09:34:49.617574 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:49.617494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl" event={"ID":"dbce62e5-6932-4ab5-b5aa-3525dc3a2398","Type":"ContainerStarted","Data":"1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3"}
Sep 15 09:34:49.618019 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:49.617598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:34:49.634454 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:49.634408    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl" podStartSLOduration=1.984836584 podStartE2EDuration="2.634393578s" totalImagesPullingTime="649.556994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:34:49.632901186 +0000 UTC m=+1253.462693090" watchObservedRunningTime="2026-09-15 09:34:49.634393578 +0000 UTC m=+1253.464185481"
Sep 15 09:34:55.509455 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:55.509419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:34:55.627792 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:34:55.627763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:35:18.283429 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.283397    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"]
Sep 15 09:35:18.288435 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.288405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.290420 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.290382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e76bac\"/\"distinct-w0-e76bac-kserve-self-signed-certs\""
Sep 15 09:35:18.294867 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.294738    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"]
Sep 15 09:35:18.353700 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.353672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-dshm\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.353813 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.353702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-model-cache\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.353813 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.353774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5414c4-2153-41a8-ba76-517bf3731bf1-tls-certs\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.353813 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.353806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-tmp-dir\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.353931 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.353874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ssr5\" (UniqueName: \"kubernetes.io/projected/ea5414c4-2153-41a8-ba76-517bf3731bf1-kube-api-access-2ssr5\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.353931 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.353926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-home\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.454743 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.454719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5414c4-2153-41a8-ba76-517bf3731bf1-tls-certs\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.454844 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.454750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-tmp-dir\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.454844 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.454783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ssr5\" (UniqueName: \"kubernetes.io/projected/ea5414c4-2153-41a8-ba76-517bf3731bf1-kube-api-access-2ssr5\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.454844 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.454816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-home\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.454981 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.454857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-dshm\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.454981 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.454873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-model-cache\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.455141 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.455116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-tmp-dir\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.455302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.455279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-model-cache\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.455423 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.455184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-home\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.457126 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.457099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-dshm\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.457381 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.457364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5414c4-2153-41a8-ba76-517bf3731bf1-tls-certs\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.464625 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.464602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ssr5\" (UniqueName: \"kubernetes.io/projected/ea5414c4-2153-41a8-ba76-517bf3731bf1-kube-api-access-2ssr5\") pod \"distinct-w0-e76bac-kserve-745989754c-jn5dq\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") " pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.599071 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.599003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:18.868796 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:18.868772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"]
Sep 15 09:35:18.870452 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:35:18.870426    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea5414c4_2153_41a8_ba76_517bf3731bf1.slice/crio-e83302ffd89d2698885ee42fc583aeb8aeaf98422e0802e415aa4066cac6c998 WatchSource:0}: Error finding container e83302ffd89d2698885ee42fc583aeb8aeaf98422e0802e415aa4066cac6c998: Status 404 returned error can't find the container with id e83302ffd89d2698885ee42fc583aeb8aeaf98422e0802e415aa4066cac6c998
Sep 15 09:35:19.769330 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:19.769293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq" event={"ID":"ea5414c4-2153-41a8-ba76-517bf3731bf1","Type":"ContainerStarted","Data":"e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b"}
Sep 15 09:35:19.769740 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:19.769352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq" event={"ID":"ea5414c4-2153-41a8-ba76-517bf3731bf1","Type":"ContainerStarted","Data":"e83302ffd89d2698885ee42fc583aeb8aeaf98422e0802e415aa4066cac6c998"}
Sep 15 09:35:19.783533 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:19.783457    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq" podStartSLOduration=1.783436944 podStartE2EDuration="1.783436944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:35:19.78330941 +0000 UTC m=+1283.613101314" watchObservedRunningTime="2026-09-15 09:35:19.783436944 +0000 UTC m=+1283.613228849"
Sep 15 09:35:28.599820 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:28.599782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:28.600215 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:28.599863    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:28.611929 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:28.611909    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:28.824462 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:28.824425    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:35:35.697087 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:35.697049    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:35:50.804045 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.804011    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"]
Sep 15 09:35:50.807963 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.807942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.810839 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.810813    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-312334ca\"/\"e2e-lifecycle-model-312334ca-kserve-self-signed-certs\""
Sep 15 09:35:50.810959 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.810813    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-312334ca\"/\"kube-root-ca.crt\""
Sep 15 09:35:50.810959 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.810854    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-312334ca\"/\"openshift-service-ca.crt\""
Sep 15 09:35:50.810959 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.810813    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-312334ca\"/\"default-dockercfg-95fph\""
Sep 15 09:35:50.818756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.818734    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"]
Sep 15 09:35:50.862864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.862840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqdvn\" (UniqueName: \"kubernetes.io/projected/334c5f15-cb22-4f26-ac58-a0c184e19371-kube-api-access-xqdvn\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.862976 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.862887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-tmp-dir\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.862976 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.862958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334c5f15-cb22-4f26-ac58-a0c184e19371-tls-certs\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.863050 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.862985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-dshm\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.863050 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.863034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-model-cache\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.863119 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.863062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-home\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.964169 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.964140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-tmp-dir\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.964272 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.964203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334c5f15-cb22-4f26-ac58-a0c184e19371-tls-certs\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.964272 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.964243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-dshm\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.964384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.964277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-model-cache\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.964384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.964314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-home\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.964503 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.964422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqdvn\" (UniqueName: \"kubernetes.io/projected/334c5f15-cb22-4f26-ac58-a0c184e19371-kube-api-access-xqdvn\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.964621 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.964602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-tmp-dir\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.964687 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.964650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-model-cache\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.964739 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.964721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-home\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.966453 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.966422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-dshm\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.966691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.966676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334c5f15-cb22-4f26-ac58-a0c184e19371-tls-certs\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:50.971437 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:50.971420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqdvn\" (UniqueName: \"kubernetes.io/projected/334c5f15-cb22-4f26-ac58-a0c184e19371-kube-api-access-xqdvn\") pod \"e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") " pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:51.118787 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:51.118723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:35:51.388607 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:51.388581    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"]
Sep 15 09:35:51.390744 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:35:51.390716    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod334c5f15_cb22_4f26_ac58_a0c184e19371.slice/crio-b4383d7cfe15e988fd0a23dee19da9b08087a87bc1c6e89f063289ba59a84bf3 WatchSource:0}: Error finding container b4383d7cfe15e988fd0a23dee19da9b08087a87bc1c6e89f063289ba59a84bf3: Status 404 returned error can't find the container with id b4383d7cfe15e988fd0a23dee19da9b08087a87bc1c6e89f063289ba59a84bf3
Sep 15 09:35:51.938151 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:51.938116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6" event={"ID":"334c5f15-cb22-4f26-ac58-a0c184e19371","Type":"ContainerStarted","Data":"5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1"}
Sep 15 09:35:51.938151 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:51.938163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6" event={"ID":"334c5f15-cb22-4f26-ac58-a0c184e19371","Type":"ContainerStarted","Data":"b4383d7cfe15e988fd0a23dee19da9b08087a87bc1c6e89f063289ba59a84bf3"}
Sep 15 09:35:51.957991 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:35:51.957940    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6" podStartSLOduration=1.957925556 podStartE2EDuration="1.957925556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:35:51.956244221 +0000 UTC m=+1315.786036135" watchObservedRunningTime="2026-09-15 09:35:51.957925556 +0000 UTC m=+1315.787717459"
Sep 15 09:36:01.119248 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:01.119213    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:36:01.119652 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:01.119307    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:36:01.131600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:01.131574    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:36:02.004430 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:02.004400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:36:08.507369 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:08.507313    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:36:14.189582 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:14.189550    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"]
Sep 15 09:36:19.622465 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.622380    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"]
Sep 15 09:36:19.622920 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.622627    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx" podUID="2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" containerName="maas-api" containerID="cri-o://b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76" gracePeriod=30
Sep 15 09:36:19.892586 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.892541    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:36:19.960674 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.960634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn7dw\" (UniqueName: \"kubernetes.io/projected/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-kube-api-access-bn7dw\") pod \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") "
Sep 15 09:36:19.960860 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.960730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-metrics-certs\") pod \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") "
Sep 15 09:36:19.960860 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.960832    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-maas-api-tls\") pod \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") "
Sep 15 09:36:19.960936 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.960893    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-openshift-service-ca\") pod \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\" (UID: \"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc\") "
Sep 15 09:36:19.961261 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.961226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" (UID: "2cc1c1f0-bac5-4ad4-b618-34afa2df2afc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:36:19.961387 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.961313    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:19.963063 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.963033    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" (UID: "2cc1c1f0-bac5-4ad4-b618-34afa2df2afc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:36:19.963511 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.963493    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" (UID: "2cc1c1f0-bac5-4ad4-b618-34afa2df2afc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:36:19.963578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:19.963520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-kube-api-access-bn7dw" (OuterVolumeSpecName: "kube-api-access-bn7dw") pod "2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" (UID: "2cc1c1f0-bac5-4ad4-b618-34afa2df2afc"). InnerVolumeSpecName "kube-api-access-bn7dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:36:20.062698 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.062659    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:20.062698 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.062695    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:20.062698 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.062704    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn7dw\" (UniqueName: \"kubernetes.io/projected/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc-kube-api-access-bn7dw\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:20.091773 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.091738    2579 generic.go:356] "Generic (PLEG): container finished" podID="2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" containerID="b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76" exitCode=0
Sep 15 09:36:20.091964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.091786    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"
Sep 15 09:36:20.091964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.091818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx" event={"ID":"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc","Type":"ContainerDied","Data":"b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76"}
Sep 15 09:36:20.091964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.091873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx" event={"ID":"2cc1c1f0-bac5-4ad4-b618-34afa2df2afc","Type":"ContainerDied","Data":"7031f0819546c92b7ef04bfe056667a24444cd669131ba44005110405020f054"}
Sep 15 09:36:20.091964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.091906    2579 scope.go:117] "RemoveContainer" containerID="b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76"
Sep 15 09:36:20.101932 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.101909    2579 scope.go:117] "RemoveContainer" containerID="b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76"
Sep 15 09:36:20.102275 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:36:20.102251    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76\": container with ID starting with b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76 not found: ID does not exist" containerID="b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76"
Sep 15 09:36:20.102327 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.102287    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76"} err="failed to get container status \"b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76\": rpc error: code = NotFound desc = could not find container \"b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76\": container with ID starting with b0c71b815d4c6bad4d79bab103ef15f597e1d0e24cc53cb1fa8995fdffcdae76 not found: ID does not exist"
Sep 15 09:36:20.128407 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.128298    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"]
Sep 15 09:36:20.134690 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.134656    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a31d6-6dd9d65575-ng7dx"]
Sep 15 09:36:20.675609 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:20.675578    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" path="/var/lib/kubelet/pods/2cc1c1f0-bac5-4ad4-b618-34afa2df2afc/volumes"
Sep 15 09:36:29.209118 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.209074    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll" podUID="6920dbda-ab79-4102-b999-040ce31c9e9e" containerName="main" containerID="cri-o://15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140" gracePeriod=45
Sep 15 09:36:29.473165 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.473106    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:36:29.555438 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.555410    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-model-cache\") pod \"6920dbda-ab79-4102-b999-040ce31c9e9e\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") "
Sep 15 09:36:29.555612 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.555493    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6920dbda-ab79-4102-b999-040ce31c9e9e-tls-certs\") pod \"6920dbda-ab79-4102-b999-040ce31c9e9e\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") "
Sep 15 09:36:29.555612 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.555535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-dshm\") pod \"6920dbda-ab79-4102-b999-040ce31c9e9e\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") "
Sep 15 09:36:29.555612 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.555568    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-home\") pod \"6920dbda-ab79-4102-b999-040ce31c9e9e\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") "
Sep 15 09:36:29.555787 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.555723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64r2g\" (UniqueName: \"kubernetes.io/projected/6920dbda-ab79-4102-b999-040ce31c9e9e-kube-api-access-64r2g\") pod \"6920dbda-ab79-4102-b999-040ce31c9e9e\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") "
Sep 15 09:36:29.555787 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.555776    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-tmp-dir\") pod \"6920dbda-ab79-4102-b999-040ce31c9e9e\" (UID: \"6920dbda-ab79-4102-b999-040ce31c9e9e\") "
Sep 15 09:36:29.555909 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.555703    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6920dbda-ab79-4102-b999-040ce31c9e9e" (UID: "6920dbda-ab79-4102-b999-040ce31c9e9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:36:29.555909 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.555773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-home" (OuterVolumeSpecName: "home") pod "6920dbda-ab79-4102-b999-040ce31c9e9e" (UID: "6920dbda-ab79-4102-b999-040ce31c9e9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:36:29.556022 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.555943    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6920dbda-ab79-4102-b999-040ce31c9e9e" (UID: "6920dbda-ab79-4102-b999-040ce31c9e9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:36:29.556198 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.556174    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:29.556198 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.556191    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:29.556198 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.556201    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:29.557811 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.557784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6920dbda-ab79-4102-b999-040ce31c9e9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6920dbda-ab79-4102-b999-040ce31c9e9e" (UID: "6920dbda-ab79-4102-b999-040ce31c9e9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:36:29.557899 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.557826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6920dbda-ab79-4102-b999-040ce31c9e9e-kube-api-access-64r2g" (OuterVolumeSpecName: "kube-api-access-64r2g") pod "6920dbda-ab79-4102-b999-040ce31c9e9e" (UID: "6920dbda-ab79-4102-b999-040ce31c9e9e"). InnerVolumeSpecName "kube-api-access-64r2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:36:29.557899 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.557847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-dshm" (OuterVolumeSpecName: "dshm") pod "6920dbda-ab79-4102-b999-040ce31c9e9e" (UID: "6920dbda-ab79-4102-b999-040ce31c9e9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:36:29.657193 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.657162    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6920dbda-ab79-4102-b999-040ce31c9e9e-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:29.657193 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.657187    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6920dbda-ab79-4102-b999-040ce31c9e9e-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:29.657193 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:29.657196    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64r2g\" (UniqueName: \"kubernetes.io/projected/6920dbda-ab79-4102-b999-040ce31c9e9e-kube-api-access-64r2g\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:30.147531 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.147499    2579 generic.go:356] "Generic (PLEG): container finished" podID="6920dbda-ab79-4102-b999-040ce31c9e9e" containerID="15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140" exitCode=0
Sep 15 09:36:30.147710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.147539    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"
Sep 15 09:36:30.147710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.147550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll" event={"ID":"6920dbda-ab79-4102-b999-040ce31c9e9e","Type":"ContainerDied","Data":"15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140"}
Sep 15 09:36:30.147710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.147581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll" event={"ID":"6920dbda-ab79-4102-b999-040ce31c9e9e","Type":"ContainerDied","Data":"216209ca1a07865bbe9620e5ed40b28a28d5af42fea52a6eddb0c4ca9c15dd79"}
Sep 15 09:36:30.147710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.147597    2579 scope.go:117] "RemoveContainer" containerID="15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140"
Sep 15 09:36:30.156824 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.156806    2579 scope.go:117] "RemoveContainer" containerID="15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140"
Sep 15 09:36:30.157100 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:36:30.157081    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140\": container with ID starting with 15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140 not found: ID does not exist" containerID="15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140"
Sep 15 09:36:30.157164 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.157115    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140"} err="failed to get container status \"15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140\": rpc error: code = NotFound desc = could not find container \"15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140\": container with ID starting with 15f94f3844255c3789579193e6d424961a11486d67615dcb3ec459df41709140 not found: ID does not exist"
Sep 15 09:36:30.167253 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.167229    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"]
Sep 15 09:36:30.171059 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.171035    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0e280a/test-model-0e280a-kserve-775594b5c7-h78ll"]
Sep 15 09:36:30.675800 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:30.675767    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6920dbda-ab79-4102-b999-040ce31c9e9e" path="/var/lib/kubelet/pods/6920dbda-ab79-4102-b999-040ce31c9e9e/volumes"
Sep 15 09:36:36.843240 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:36.843196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"]
Sep 15 09:36:36.843720 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:36.843490    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94" podUID="444b23db-7252-4d5c-a339-c52a3c28015f" containerName="maas-api" containerID="cri-o://230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216" gracePeriod=30
Sep 15 09:36:37.103732 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.103656    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:36:37.187557 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.187517    2579 generic.go:356] "Generic (PLEG): container finished" podID="444b23db-7252-4d5c-a339-c52a3c28015f" containerID="230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216" exitCode=0
Sep 15 09:36:37.187557 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.187558    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"
Sep 15 09:36:37.187788 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.187596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94" event={"ID":"444b23db-7252-4d5c-a339-c52a3c28015f","Type":"ContainerDied","Data":"230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216"}
Sep 15 09:36:37.187788 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.187654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94" event={"ID":"444b23db-7252-4d5c-a339-c52a3c28015f","Type":"ContainerDied","Data":"0fe36da74ef7ef0e1d0571b821b43092398bf9c4bfb3a7e47bb0f76e220b9d15"}
Sep 15 09:36:37.187788 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.187682    2579 scope.go:117] "RemoveContainer" containerID="230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216"
Sep 15 09:36:37.197368 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.197317    2579 scope.go:117] "RemoveContainer" containerID="230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216"
Sep 15 09:36:37.197721 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:36:37.197700    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216\": container with ID starting with 230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216 not found: ID does not exist" containerID="230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216"
Sep 15 09:36:37.197776 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.197733    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216"} err="failed to get container status \"230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216\": rpc error: code = NotFound desc = could not find container \"230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216\": container with ID starting with 230a098d1909e2e015bfdbf39b92e987855efe476f08af3da38873c3c82ef216 not found: ID does not exist"
Sep 15 09:36:37.231615 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.231579    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c9hq\" (UniqueName: \"kubernetes.io/projected/444b23db-7252-4d5c-a339-c52a3c28015f-kube-api-access-9c9hq\") pod \"444b23db-7252-4d5c-a339-c52a3c28015f\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") "
Sep 15 09:36:37.231789 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.231650    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444b23db-7252-4d5c-a339-c52a3c28015f-openshift-service-ca\") pod \"444b23db-7252-4d5c-a339-c52a3c28015f\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") "
Sep 15 09:36:37.231789 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.231671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-maas-api-tls\") pod \"444b23db-7252-4d5c-a339-c52a3c28015f\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") "
Sep 15 09:36:37.231789 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.231775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-metrics-certs\") pod \"444b23db-7252-4d5c-a339-c52a3c28015f\" (UID: \"444b23db-7252-4d5c-a339-c52a3c28015f\") "
Sep 15 09:36:37.232109 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.232075    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444b23db-7252-4d5c-a339-c52a3c28015f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "444b23db-7252-4d5c-a339-c52a3c28015f" (UID: "444b23db-7252-4d5c-a339-c52a3c28015f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:36:37.232258 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.232238    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444b23db-7252-4d5c-a339-c52a3c28015f-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:37.233868 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.233841    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "444b23db-7252-4d5c-a339-c52a3c28015f" (UID: "444b23db-7252-4d5c-a339-c52a3c28015f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:36:37.234277 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.234252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "444b23db-7252-4d5c-a339-c52a3c28015f" (UID: "444b23db-7252-4d5c-a339-c52a3c28015f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:36:37.234380 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.234280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444b23db-7252-4d5c-a339-c52a3c28015f-kube-api-access-9c9hq" (OuterVolumeSpecName: "kube-api-access-9c9hq") pod "444b23db-7252-4d5c-a339-c52a3c28015f" (UID: "444b23db-7252-4d5c-a339-c52a3c28015f"). InnerVolumeSpecName "kube-api-access-9c9hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:36:37.333061 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.333005    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c9hq\" (UniqueName: \"kubernetes.io/projected/444b23db-7252-4d5c-a339-c52a3c28015f-kube-api-access-9c9hq\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:37.333061 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.333056    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:37.333061 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.333067    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444b23db-7252-4d5c-a339-c52a3c28015f-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:36:37.509676 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.509639    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"]
Sep 15 09:36:37.515130 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:37.515096    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e280a-b66487bf8-7pf94"]
Sep 15 09:36:38.675462 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:38.675430    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="444b23db-7252-4d5c-a339-c52a3c28015f" path="/var/lib/kubelet/pods/444b23db-7252-4d5c-a339-c52a3c28015f/volumes"
Sep 15 09:36:44.704786 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:36:44.704746    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:37:05.529654 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.529617    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"]
Sep 15 09:37:05.530233 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.529900    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl" podUID="dbce62e5-6932-4ab5-b5aa-3525dc3a2398" containerName="maas-api" containerID="cri-o://1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3" gracePeriod=30
Sep 15 09:37:05.623579 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.623535    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl" podUID="dbce62e5-6932-4ab5-b5aa-3525dc3a2398" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.58:8443/health\": dial tcp 10.132.0.58:8443: connect: connection refused"
Sep 15 09:37:05.781558 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.781491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:37:05.809421 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.809388    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-maas-api-tls\") pod \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") "
Sep 15 09:37:05.809572 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.809444    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-openshift-service-ca\") pod \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") "
Sep 15 09:37:05.809572 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.809537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4czp\" (UniqueName: \"kubernetes.io/projected/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-kube-api-access-f4czp\") pod \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") "
Sep 15 09:37:05.809572 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.809560    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-metrics-certs\") pod \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\" (UID: \"dbce62e5-6932-4ab5-b5aa-3525dc3a2398\") "
Sep 15 09:37:05.809865 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.809822    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbce62e5-6932-4ab5-b5aa-3525dc3a2398" (UID: "dbce62e5-6932-4ab5-b5aa-3525dc3a2398"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:37:05.810014 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.809952    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:05.811588 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.811563    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-kube-api-access-f4czp" (OuterVolumeSpecName: "kube-api-access-f4czp") pod "dbce62e5-6932-4ab5-b5aa-3525dc3a2398" (UID: "dbce62e5-6932-4ab5-b5aa-3525dc3a2398"). InnerVolumeSpecName "kube-api-access-f4czp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:37:05.811700 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.811604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dbce62e5-6932-4ab5-b5aa-3525dc3a2398" (UID: "dbce62e5-6932-4ab5-b5aa-3525dc3a2398"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:37:05.811700 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.811679    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbce62e5-6932-4ab5-b5aa-3525dc3a2398" (UID: "dbce62e5-6932-4ab5-b5aa-3525dc3a2398"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:37:05.910606 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.910577    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4czp\" (UniqueName: \"kubernetes.io/projected/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-kube-api-access-f4czp\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:05.910728 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.910608    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:05.910728 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:05.910623    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbce62e5-6932-4ab5-b5aa-3525dc3a2398-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:06.345214 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.345175    2579 generic.go:356] "Generic (PLEG): container finished" podID="dbce62e5-6932-4ab5-b5aa-3525dc3a2398" containerID="1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3" exitCode=0
Sep 15 09:37:06.345407 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.345228    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"
Sep 15 09:37:06.345407 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.345257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl" event={"ID":"dbce62e5-6932-4ab5-b5aa-3525dc3a2398","Type":"ContainerDied","Data":"1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3"}
Sep 15 09:37:06.345407 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.345321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl" event={"ID":"dbce62e5-6932-4ab5-b5aa-3525dc3a2398","Type":"ContainerDied","Data":"0af4d355c9be1895938ddd9c0204f91435111aba63f2f4e69a17112f03300056"}
Sep 15 09:37:06.345407 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.345356    2579 scope.go:117] "RemoveContainer" containerID="1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3"
Sep 15 09:37:06.355256 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.355239    2579 scope.go:117] "RemoveContainer" containerID="1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3"
Sep 15 09:37:06.355564 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:37:06.355542    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3\": container with ID starting with 1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3 not found: ID does not exist" containerID="1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3"
Sep 15 09:37:06.355608 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.355572    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3"} err="failed to get container status \"1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3\": rpc error: code = NotFound desc = could not find container \"1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3\": container with ID starting with 1c147a920a2578bb91651af7b1b68c5dfbfca656eba4d6728ce1d731e572b1d3 not found: ID does not exist"
Sep 15 09:37:06.372265 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.372245    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"]
Sep 15 09:37:06.379172 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.379149    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-312334ca-7fdd55cd87-dvlgl"]
Sep 15 09:37:06.675517 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:06.675485    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbce62e5-6932-4ab5-b5aa-3525dc3a2398" path="/var/lib/kubelet/pods/dbce62e5-6932-4ab5-b5aa-3525dc3a2398/volumes"
Sep 15 09:37:14.073223 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:14.073179    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"]
Sep 15 09:37:21.604749 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.604717    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"]
Sep 15 09:37:21.605221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605105    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbce62e5-6932-4ab5-b5aa-3525dc3a2398" containerName="maas-api"
Sep 15 09:37:21.605221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605117    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbce62e5-6932-4ab5-b5aa-3525dc3a2398" containerName="maas-api"
Sep 15 09:37:21.605221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605128    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6920dbda-ab79-4102-b999-040ce31c9e9e" containerName="main"
Sep 15 09:37:21.605221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605133    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6920dbda-ab79-4102-b999-040ce31c9e9e" containerName="main"
Sep 15 09:37:21.605221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605152    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="444b23db-7252-4d5c-a339-c52a3c28015f" containerName="maas-api"
Sep 15 09:37:21.605221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605158    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="444b23db-7252-4d5c-a339-c52a3c28015f" containerName="maas-api"
Sep 15 09:37:21.605221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605165    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" containerName="maas-api"
Sep 15 09:37:21.605221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605170    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" containerName="maas-api"
Sep 15 09:37:21.605221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605227    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6920dbda-ab79-4102-b999-040ce31c9e9e" containerName="main"
Sep 15 09:37:21.605756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605251    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="444b23db-7252-4d5c-a339-c52a3c28015f" containerName="maas-api"
Sep 15 09:37:21.605756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605258    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc1c1f0-bac5-4ad4-b618-34afa2df2afc" containerName="maas-api"
Sep 15 09:37:21.605756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.605265    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbce62e5-6932-4ab5-b5aa-3525dc3a2398" containerName="maas-api"
Sep 15 09:37:21.609879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.609859    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.611802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.611782    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-b283ee-serving-cert\""
Sep 15 09:37:21.624650 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.624623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"]
Sep 15 09:37:21.767673 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.767637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zc7\" (UniqueName: \"kubernetes.io/projected/a00ca908-2ddd-447b-bf38-6e010f11c279-kube-api-access-65zc7\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.767852 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.767719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.767852 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.767761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ca908-2ddd-447b-bf38-6e010f11c279-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.767852 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.767818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.869040 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.868967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.869040 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.869013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ca908-2ddd-447b-bf38-6e010f11c279-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.869040 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.869040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.869289 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.869076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zc7\" (UniqueName: \"kubernetes.io/projected/a00ca908-2ddd-447b-bf38-6e010f11c279-kube-api-access-65zc7\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.869743 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.869720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ca908-2ddd-447b-bf38-6e010f11c279-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.871441 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.871413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.871529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.871472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.878806 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.878781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zc7\" (UniqueName: \"kubernetes.io/projected/a00ca908-2ddd-447b-bf38-6e010f11c279-kube-api-access-65zc7\") pod \"maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:21.925014 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:21.924976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:22.194827 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:22.194770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"]
Sep 15 09:37:22.197330 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:37:22.197302    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00ca908_2ddd_447b_bf38_6e010f11c279.slice/crio-dc663957f657d123996748e849fc00806095b82af6b5c3d9fc9c3d44d6214d58 WatchSource:0}: Error finding container dc663957f657d123996748e849fc00806095b82af6b5c3d9fc9c3d44d6214d58: Status 404 returned error can't find the container with id dc663957f657d123996748e849fc00806095b82af6b5c3d9fc9c3d44d6214d58
Sep 15 09:37:22.198913 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:22.198892    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:37:22.431858 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:22.431823    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp" event={"ID":"a00ca908-2ddd-447b-bf38-6e010f11c279","Type":"ContainerStarted","Data":"dc663957f657d123996748e849fc00806095b82af6b5c3d9fc9c3d44d6214d58"}
Sep 15 09:37:23.438603 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:23.438571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp" event={"ID":"a00ca908-2ddd-447b-bf38-6e010f11c279","Type":"ContainerStarted","Data":"f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946"}
Sep 15 09:37:23.439063 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:23.438657    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:23.456921 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:23.456877    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp" podStartSLOduration=2.1603204480000002 podStartE2EDuration="2.456865297s" totalImagesPullingTime="296.544849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:37:23.454417301 +0000 UTC m=+1407.284209203" watchObservedRunningTime="2026-09-15 09:37:23.456865297 +0000 UTC m=+1407.286657212"
Sep 15 09:37:29.410206 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.410165    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6" podUID="334c5f15-cb22-4f26-ac58-a0c184e19371" containerName="main" containerID="cri-o://5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1" gracePeriod=45
Sep 15 09:37:29.448892 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.448870    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:37:29.661160 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.661107    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:37:29.845788 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.845758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334c5f15-cb22-4f26-ac58-a0c184e19371-tls-certs\") pod \"334c5f15-cb22-4f26-ac58-a0c184e19371\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") "
Sep 15 09:37:29.845939 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.845802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-dshm\") pod \"334c5f15-cb22-4f26-ac58-a0c184e19371\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") "
Sep 15 09:37:29.845939 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.845843    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-model-cache\") pod \"334c5f15-cb22-4f26-ac58-a0c184e19371\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") "
Sep 15 09:37:29.846050 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.846016    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqdvn\" (UniqueName: \"kubernetes.io/projected/334c5f15-cb22-4f26-ac58-a0c184e19371-kube-api-access-xqdvn\") pod \"334c5f15-cb22-4f26-ac58-a0c184e19371\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") "
Sep 15 09:37:29.846087 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.846069    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-tmp-dir\") pod \"334c5f15-cb22-4f26-ac58-a0c184e19371\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") "
Sep 15 09:37:29.846141 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.846123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-home\") pod \"334c5f15-cb22-4f26-ac58-a0c184e19371\" (UID: \"334c5f15-cb22-4f26-ac58-a0c184e19371\") "
Sep 15 09:37:29.846224 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.846067    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-model-cache" (OuterVolumeSpecName: "model-cache") pod "334c5f15-cb22-4f26-ac58-a0c184e19371" (UID: "334c5f15-cb22-4f26-ac58-a0c184e19371"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:37:29.846329 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.846271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "334c5f15-cb22-4f26-ac58-a0c184e19371" (UID: "334c5f15-cb22-4f26-ac58-a0c184e19371"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:37:29.846503 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.846355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-home" (OuterVolumeSpecName: "home") pod "334c5f15-cb22-4f26-ac58-a0c184e19371" (UID: "334c5f15-cb22-4f26-ac58-a0c184e19371"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:37:29.846639 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.846620    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:29.846701 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.846646    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:29.846701 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.846662    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:29.848100 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.848078    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334c5f15-cb22-4f26-ac58-a0c184e19371-kube-api-access-xqdvn" (OuterVolumeSpecName: "kube-api-access-xqdvn") pod "334c5f15-cb22-4f26-ac58-a0c184e19371" (UID: "334c5f15-cb22-4f26-ac58-a0c184e19371"). InnerVolumeSpecName "kube-api-access-xqdvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:37:29.848280 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.848254    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-dshm" (OuterVolumeSpecName: "dshm") pod "334c5f15-cb22-4f26-ac58-a0c184e19371" (UID: "334c5f15-cb22-4f26-ac58-a0c184e19371"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:37:29.848280 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.848262    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334c5f15-cb22-4f26-ac58-a0c184e19371-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "334c5f15-cb22-4f26-ac58-a0c184e19371" (UID: "334c5f15-cb22-4f26-ac58-a0c184e19371"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:37:29.947514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.947462    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334c5f15-cb22-4f26-ac58-a0c184e19371-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:29.947514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.947481    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334c5f15-cb22-4f26-ac58-a0c184e19371-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:29.947514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:29.947492    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqdvn\" (UniqueName: \"kubernetes.io/projected/334c5f15-cb22-4f26-ac58-a0c184e19371-kube-api-access-xqdvn\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:30.078529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.078490    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"]
Sep 15 09:37:30.476248 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.476218    2579 generic.go:356] "Generic (PLEG): container finished" podID="334c5f15-cb22-4f26-ac58-a0c184e19371" containerID="5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1" exitCode=0
Sep 15 09:37:30.476636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.476260    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"
Sep 15 09:37:30.476636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.476301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6" event={"ID":"334c5f15-cb22-4f26-ac58-a0c184e19371","Type":"ContainerDied","Data":"5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1"}
Sep 15 09:37:30.476636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.476352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6" event={"ID":"334c5f15-cb22-4f26-ac58-a0c184e19371","Type":"ContainerDied","Data":"b4383d7cfe15e988fd0a23dee19da9b08087a87bc1c6e89f063289ba59a84bf3"}
Sep 15 09:37:30.476636 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.476371    2579 scope.go:117] "RemoveContainer" containerID="5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1"
Sep 15 09:37:30.485878 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.485862    2579 scope.go:117] "RemoveContainer" containerID="5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1"
Sep 15 09:37:30.486158 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:37:30.486140    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1\": container with ID starting with 5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1 not found: ID does not exist" containerID="5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1"
Sep 15 09:37:30.486202 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.486170    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1"} err="failed to get container status \"5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1\": rpc error: code = NotFound desc = could not find container \"5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1\": container with ID starting with 5c8f5660ff7853bca9d2b281c2a2efddeeccdbe1a28d1224b9436cb28d7842e1 not found: ID does not exist"
Sep 15 09:37:30.496842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.496819    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"]
Sep 15 09:37:30.502565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.502546    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-312334ca/e2e-lifecycle-model-312334ca-kserve-7bb9b98459-57dp6"]
Sep 15 09:37:30.675494 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:30.675470    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334c5f15-cb22-4f26-ac58-a0c184e19371" path="/var/lib/kubelet/pods/334c5f15-cb22-4f26-ac58-a0c184e19371/volumes"
Sep 15 09:37:31.106949 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:31.106916    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:37:45.098076 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.098031    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb" podUID="80f35a0d-166c-4577-b169-d811e61448ad" containerName="main" containerID="cri-o://ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e" gracePeriod=45
Sep 15 09:37:45.342016 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.341992    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:37:45.401192 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401106    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgdqq\" (UniqueName: \"kubernetes.io/projected/80f35a0d-166c-4577-b169-d811e61448ad-kube-api-access-sgdqq\") pod \"80f35a0d-166c-4577-b169-d811e61448ad\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") "
Sep 15 09:37:45.401192 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-model-cache\") pod \"80f35a0d-166c-4577-b169-d811e61448ad\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") "
Sep 15 09:37:45.401433 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401200    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-dshm\") pod \"80f35a0d-166c-4577-b169-d811e61448ad\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") "
Sep 15 09:37:45.401433 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-home\") pod \"80f35a0d-166c-4577-b169-d811e61448ad\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") "
Sep 15 09:37:45.401433 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401288    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35a0d-166c-4577-b169-d811e61448ad-tls-certs\") pod \"80f35a0d-166c-4577-b169-d811e61448ad\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") "
Sep 15 09:37:45.401433 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-tmp-dir\") pod \"80f35a0d-166c-4577-b169-d811e61448ad\" (UID: \"80f35a0d-166c-4577-b169-d811e61448ad\") "
Sep 15 09:37:45.401626 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401446    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "80f35a0d-166c-4577-b169-d811e61448ad" (UID: "80f35a0d-166c-4577-b169-d811e61448ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:37:45.401626 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401557    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-home" (OuterVolumeSpecName: "home") pod "80f35a0d-166c-4577-b169-d811e61448ad" (UID: "80f35a0d-166c-4577-b169-d811e61448ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:37:45.401719 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80f35a0d-166c-4577-b169-d811e61448ad" (UID: "80f35a0d-166c-4577-b169-d811e61448ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:37:45.401798 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401775    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:45.401859 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401801    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:45.401859 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.401816    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:45.403497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.403476    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f35a0d-166c-4577-b169-d811e61448ad-kube-api-access-sgdqq" (OuterVolumeSpecName: "kube-api-access-sgdqq") pod "80f35a0d-166c-4577-b169-d811e61448ad" (UID: "80f35a0d-166c-4577-b169-d811e61448ad"). InnerVolumeSpecName "kube-api-access-sgdqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:37:45.403497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.403489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-dshm" (OuterVolumeSpecName: "dshm") pod "80f35a0d-166c-4577-b169-d811e61448ad" (UID: "80f35a0d-166c-4577-b169-d811e61448ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:37:45.403619 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.403529    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f35a0d-166c-4577-b169-d811e61448ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80f35a0d-166c-4577-b169-d811e61448ad" (UID: "80f35a0d-166c-4577-b169-d811e61448ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:37:45.502772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.502738    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgdqq\" (UniqueName: \"kubernetes.io/projected/80f35a0d-166c-4577-b169-d811e61448ad-kube-api-access-sgdqq\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:45.502772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.502771    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35a0d-166c-4577-b169-d811e61448ad-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:45.502772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.502780    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35a0d-166c-4577-b169-d811e61448ad-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:37:45.555217 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.555186    2579 generic.go:356] "Generic (PLEG): container finished" podID="80f35a0d-166c-4577-b169-d811e61448ad" containerID="ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e" exitCode=0
Sep 15 09:37:45.555401 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.555231    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"
Sep 15 09:37:45.555401 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.555278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb" event={"ID":"80f35a0d-166c-4577-b169-d811e61448ad","Type":"ContainerDied","Data":"ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e"}
Sep 15 09:37:45.555401 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.555319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb" event={"ID":"80f35a0d-166c-4577-b169-d811e61448ad","Type":"ContainerDied","Data":"9ee02ec044b5a66acf75bd925afe4faae1105058890ca5e55894658f81b31d7a"}
Sep 15 09:37:45.555401 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.555361    2579 scope.go:117] "RemoveContainer" containerID="ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e"
Sep 15 09:37:45.564563 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.564546    2579 scope.go:117] "RemoveContainer" containerID="ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e"
Sep 15 09:37:45.564838 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:37:45.564818    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e\": container with ID starting with ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e not found: ID does not exist" containerID="ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e"
Sep 15 09:37:45.564892 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.564845    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e"} err="failed to get container status \"ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e\": rpc error: code = NotFound desc = could not find container \"ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e\": container with ID starting with ec08aee06a3b57c4add7da4266909778187104575de125ab9a3f8b7c5199e68e not found: ID does not exist"
Sep 15 09:37:45.574961 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.574935    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"]
Sep 15 09:37:45.581326 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:45.580688    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/e2e-conflict-a-0b195997-kserve-8559cdd67f-c8cdb"]
Sep 15 09:37:46.675640 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:37:46.675611    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f35a0d-166c-4577-b169-d811e61448ad" path="/var/lib/kubelet/pods/80f35a0d-166c-4577-b169-d811e61448ad/volumes"
Sep 15 09:38:01.596132 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.596098    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"]
Sep 15 09:38:01.596969 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.596939    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334c5f15-cb22-4f26-ac58-a0c184e19371" containerName="main"
Sep 15 09:38:01.596969 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.596965    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="334c5f15-cb22-4f26-ac58-a0c184e19371" containerName="main"
Sep 15 09:38:01.597188 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.596987    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f35a0d-166c-4577-b169-d811e61448ad" containerName="main"
Sep 15 09:38:01.597188 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.596996    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f35a0d-166c-4577-b169-d811e61448ad" containerName="main"
Sep 15 09:38:01.597188 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.597076    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f35a0d-166c-4577-b169-d811e61448ad" containerName="main"
Sep 15 09:38:01.597188 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.597086    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="334c5f15-cb22-4f26-ac58-a0c184e19371" containerName="main"
Sep 15 09:38:01.601632 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.601609    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.605411 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.605381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c183a5\"/\"distinct-w4-c183a5-kserve-self-signed-certs\""
Sep 15 09:38:01.610711 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.610688    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"]
Sep 15 09:38:01.765003 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.764950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-dshm\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.765003 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.764999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsxk7\" (UniqueName: \"kubernetes.io/projected/a58994ca-7521-488c-8e4a-09d603fc9968-kube-api-access-gsxk7\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.765286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.765032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-model-cache\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.765286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.765109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-tmp-dir\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.765286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.765151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58994ca-7521-488c-8e4a-09d603fc9968-tls-certs\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.765496 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.765318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-home\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.866432 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.866287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-tmp-dir\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.866432 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.866380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58994ca-7521-488c-8e4a-09d603fc9968-tls-certs\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.866432 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.866437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-home\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.866730 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.866478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-dshm\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.866730 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.866494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsxk7\" (UniqueName: \"kubernetes.io/projected/a58994ca-7521-488c-8e4a-09d603fc9968-kube-api-access-gsxk7\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.866730 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.866514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-model-cache\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.866886 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.866798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-tmp-dir\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.866886 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.866859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-home\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.866986 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.866965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-model-cache\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.868777 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.868757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-dshm\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.868947 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.868930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58994ca-7521-488c-8e4a-09d603fc9968-tls-certs\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.875653 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.875624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsxk7\" (UniqueName: \"kubernetes.io/projected/a58994ca-7521-488c-8e4a-09d603fc9968-kube-api-access-gsxk7\") pod \"distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") " pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:01.913232 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:01.913196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:02.192008 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:02.191972    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"]
Sep 15 09:38:02.193532 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:38:02.193502    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda58994ca_7521_488c_8e4a_09d603fc9968.slice/crio-49033a9dd7afec61a98ad6dc7fb5270530a833f7162b81427757ad8dfb36869d WatchSource:0}: Error finding container 49033a9dd7afec61a98ad6dc7fb5270530a833f7162b81427757ad8dfb36869d: Status 404 returned error can't find the container with id 49033a9dd7afec61a98ad6dc7fb5270530a833f7162b81427757ad8dfb36869d
Sep 15 09:38:02.646612 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:02.646518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr" event={"ID":"a58994ca-7521-488c-8e4a-09d603fc9968","Type":"ContainerStarted","Data":"6b25a24691f5e29fb00e973152bd3588f587a727080323ba060065061074f6ef"}
Sep 15 09:38:02.646612 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:02.646562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr" event={"ID":"a58994ca-7521-488c-8e4a-09d603fc9968","Type":"ContainerStarted","Data":"49033a9dd7afec61a98ad6dc7fb5270530a833f7162b81427757ad8dfb36869d"}
Sep 15 09:38:02.663233 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:02.663175    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr" podStartSLOduration=1.6631574420000002 podStartE2EDuration="1.663157442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:38:02.660498877 +0000 UTC m=+1446.490290787" watchObservedRunningTime="2026-09-15 09:38:02.663157442 +0000 UTC m=+1446.492949382"
Sep 15 09:38:05.894572 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.894540    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"]
Sep 15 09:38:05.898543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.898523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:05.901790 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.901764    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-864b9a\"/\"kube-root-ca.crt\""
Sep 15 09:38:05.901916 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.901803    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-864b9a\"/\"default-dockercfg-rwnsj\""
Sep 15 09:38:05.901916 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.901812    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-864b9a\"/\"auth-test-model-864b9a-kserve-self-signed-certs\""
Sep 15 09:38:05.901916 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.901851    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-864b9a\"/\"openshift-service-ca.crt\""
Sep 15 09:38:05.907836 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.907814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-model-cache\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:05.908024 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.908001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-tmp-dir\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:05.908146 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.908063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc423024-7539-4d80-8ecc-4a41be184b31-tls-certs\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:05.908146 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.908104    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"]
Sep 15 09:38:05.908146 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.908121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-home\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:05.908315 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.908194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kr5t\" (UniqueName: \"kubernetes.io/projected/bc423024-7539-4d80-8ecc-4a41be184b31-kube-api-access-4kr5t\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:05.908315 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:05.908243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-dshm\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.009297 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.009259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-dshm\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.009480 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.009355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-model-cache\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.009480 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.009374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-tmp-dir\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.009480 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.009405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc423024-7539-4d80-8ecc-4a41be184b31-tls-certs\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.009480 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.009450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-home\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.009712 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.009493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kr5t\" (UniqueName: \"kubernetes.io/projected/bc423024-7539-4d80-8ecc-4a41be184b31-kube-api-access-4kr5t\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.009791 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.009755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-model-cache\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.009855 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.009760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-tmp-dir\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.009855 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.009826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-home\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.011431 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.011410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-dshm\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.011905 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.011882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc423024-7539-4d80-8ecc-4a41be184b31-tls-certs\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.016278 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.016250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kr5t\" (UniqueName: \"kubernetes.io/projected/bc423024-7539-4d80-8ecc-4a41be184b31-kube-api-access-4kr5t\") pod \"auth-test-model-864b9a-kserve-686f8c7b67-8qpt5\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.210067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.210030    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:06.483803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.483772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"]
Sep 15 09:38:06.484304 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:38:06.484273    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc423024_7539_4d80_8ecc_4a41be184b31.slice/crio-b03c835f8aa5073a09e5cd45cbb1e582a8751b848b94387f6e28f9c3bbedc3c5 WatchSource:0}: Error finding container b03c835f8aa5073a09e5cd45cbb1e582a8751b848b94387f6e28f9c3bbedc3c5: Status 404 returned error can't find the container with id b03c835f8aa5073a09e5cd45cbb1e582a8751b848b94387f6e28f9c3bbedc3c5
Sep 15 09:38:06.675529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.675496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5" event={"ID":"bc423024-7539-4d80-8ecc-4a41be184b31","Type":"ContainerStarted","Data":"6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da"}
Sep 15 09:38:06.675529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.675532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5" event={"ID":"bc423024-7539-4d80-8ecc-4a41be184b31","Type":"ContainerStarted","Data":"b03c835f8aa5073a09e5cd45cbb1e582a8751b848b94387f6e28f9c3bbedc3c5"}
Sep 15 09:38:06.686736 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:06.686682    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5" podStartSLOduration=1.686662756 podStartE2EDuration="1.686662756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:38:06.686104572 +0000 UTC m=+1450.515896476" watchObservedRunningTime="2026-09-15 09:38:06.686662756 +0000 UTC m=+1450.516454660"
Sep 15 09:38:10.711321 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:10.711286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:38:11.913569 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:11.913536    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:11.913569 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:11.913580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:11.926383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:11.926329    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:12.710677 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:12.710648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:38:16.210830 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:16.210793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:16.211314 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:16.210886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:16.223461 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:16.223439    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:16.732399 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:16.732367    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:38:48.707622 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:38:48.707587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:39:12.509326 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.509291    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"]
Sep 15 09:39:12.513310 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.513290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.515412 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.515389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c9d6bb-serving-cert\""
Sep 15 09:39:12.523885 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.523762    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"]
Sep 15 09:39:12.618589 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.618560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-metrics-certs\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.618717 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.618626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e386c89b-ffb6-45a3-b00e-c99c405c0303-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.618717 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.618665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96msq\" (UniqueName: \"kubernetes.io/projected/e386c89b-ffb6-45a3-b00e-c99c405c0303-kube-api-access-96msq\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.618717 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.618700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-maas-api-tls\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.719753 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.719729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e386c89b-ffb6-45a3-b00e-c99c405c0303-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.719889 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.719764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96msq\" (UniqueName: \"kubernetes.io/projected/e386c89b-ffb6-45a3-b00e-c99c405c0303-kube-api-access-96msq\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.719889 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.719801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-maas-api-tls\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.719889 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.719829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-metrics-certs\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.720376 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.720325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e386c89b-ffb6-45a3-b00e-c99c405c0303-openshift-service-ca\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.722189 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.722171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-metrics-certs\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.722293 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.722215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-maas-api-tls\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.726691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.726670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96msq\" (UniqueName: \"kubernetes.io/projected/e386c89b-ffb6-45a3-b00e-c99c405c0303-kube-api-access-96msq\") pod \"maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:12.826208 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:12.826155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:13.102706 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:13.102680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"]
Sep 15 09:39:13.104723 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:39:13.104695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode386c89b_ffb6_45a3_b00e_c99c405c0303.slice/crio-44763c295abc97a30273a9a28eac674b44b38deadbca21bce5bc7f66edc6eb79 WatchSource:0}: Error finding container 44763c295abc97a30273a9a28eac674b44b38deadbca21bce5bc7f66edc6eb79: Status 404 returned error can't find the container with id 44763c295abc97a30273a9a28eac674b44b38deadbca21bce5bc7f66edc6eb79
Sep 15 09:39:14.025673 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:14.025640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f" event={"ID":"e386c89b-ffb6-45a3-b00e-c99c405c0303","Type":"ContainerStarted","Data":"c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40"}
Sep 15 09:39:14.026128 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:14.025695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f" event={"ID":"e386c89b-ffb6-45a3-b00e-c99c405c0303","Type":"ContainerStarted","Data":"44763c295abc97a30273a9a28eac674b44b38deadbca21bce5bc7f66edc6eb79"}
Sep 15 09:39:14.026128 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:14.025721    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:14.043220 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:14.043174    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f" podStartSLOduration=1.659764707 podStartE2EDuration="2.043159305s" totalImagesPullingTime="383.394598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:39:14.040803305 +0000 UTC m=+1517.870595219" watchObservedRunningTime="2026-09-15 09:39:14.043159305 +0000 UTC m=+1517.872951207"
Sep 15 09:39:20.036709 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:20.036633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:39:24.102134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:24.102101    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:39:47.122133 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.122100    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"]
Sep 15 09:39:47.126524 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.126501    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.129043 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.129021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e029b0-serving-cert\""
Sep 15 09:39:47.135332 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.135307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"]
Sep 15 09:39:47.245424 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.245395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-metrics-certs\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.245580 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.245467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-maas-api-tls\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.245580 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.245550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf65m\" (UniqueName: \"kubernetes.io/projected/90829547-1aad-4b4b-96f2-59ec4fc2be37-kube-api-access-gf65m\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.245659 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.245607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90829547-1aad-4b4b-96f2-59ec4fc2be37-openshift-service-ca\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.347068 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.347040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90829547-1aad-4b4b-96f2-59ec4fc2be37-openshift-service-ca\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.347223 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.347106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-metrics-certs\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.347223 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.347157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-maas-api-tls\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.347223 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.347184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf65m\" (UniqueName: \"kubernetes.io/projected/90829547-1aad-4b4b-96f2-59ec4fc2be37-kube-api-access-gf65m\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.347880 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.347843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90829547-1aad-4b4b-96f2-59ec4fc2be37-openshift-service-ca\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.349638 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.349615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-maas-api-tls\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.349735 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.349711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-metrics-certs\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.354378 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.354323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf65m\" (UniqueName: \"kubernetes.io/projected/90829547-1aad-4b4b-96f2-59ec4fc2be37-kube-api-access-gf65m\") pod \"maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.441315 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.441290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:47.718746 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:47.718719    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"]
Sep 15 09:39:47.721004 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:39:47.720978    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90829547_1aad_4b4b_96f2_59ec4fc2be37.slice/crio-1a1c028062df10f0fdad7d9563b30d05b2534dc936b03e46171dd5ff447c8209 WatchSource:0}: Error finding container 1a1c028062df10f0fdad7d9563b30d05b2534dc936b03e46171dd5ff447c8209: Status 404 returned error can't find the container with id 1a1c028062df10f0fdad7d9563b30d05b2534dc936b03e46171dd5ff447c8209
Sep 15 09:39:48.208280 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:48.208244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn" event={"ID":"90829547-1aad-4b4b-96f2-59ec4fc2be37","Type":"ContainerStarted","Data":"40126d43f70c938546f136d7fa6eede5ad882885b828213fa4a1211346dc4aa5"}
Sep 15 09:39:48.208634 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:48.208296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn" event={"ID":"90829547-1aad-4b4b-96f2-59ec4fc2be37","Type":"ContainerStarted","Data":"1a1c028062df10f0fdad7d9563b30d05b2534dc936b03e46171dd5ff447c8209"}
Sep 15 09:39:48.208634 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:48.208446    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:39:48.224479 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:48.224435    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn" podStartSLOduration=0.846712073 podStartE2EDuration="1.224421019s" totalImagesPullingTime="377.708946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:39:48.221726079 +0000 UTC m=+1552.051517982" watchObservedRunningTime="2026-09-15 09:39:48.224421019 +0000 UTC m=+1552.054212922"
Sep 15 09:39:54.218248 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:39:54.218217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:40:14.109930 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:14.109897    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:40:30.684003 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:30.683972    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"]
Sep 15 09:40:30.684588 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:30.684206    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f" podUID="e386c89b-ffb6-45a3-b00e-c99c405c0303" containerName="maas-api" containerID="cri-o://c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40" gracePeriod=30
Sep 15 09:40:30.919780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:30.919747    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:40:31.045669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.045579    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-metrics-certs\") pod \"e386c89b-ffb6-45a3-b00e-c99c405c0303\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") "
Sep 15 09:40:31.045669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.045635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96msq\" (UniqueName: \"kubernetes.io/projected/e386c89b-ffb6-45a3-b00e-c99c405c0303-kube-api-access-96msq\") pod \"e386c89b-ffb6-45a3-b00e-c99c405c0303\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") "
Sep 15 09:40:31.045669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.045659    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-maas-api-tls\") pod \"e386c89b-ffb6-45a3-b00e-c99c405c0303\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") "
Sep 15 09:40:31.045947 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.045718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e386c89b-ffb6-45a3-b00e-c99c405c0303-openshift-service-ca\") pod \"e386c89b-ffb6-45a3-b00e-c99c405c0303\" (UID: \"e386c89b-ffb6-45a3-b00e-c99c405c0303\") "
Sep 15 09:40:31.046116 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.046090    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e386c89b-ffb6-45a3-b00e-c99c405c0303-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e386c89b-ffb6-45a3-b00e-c99c405c0303" (UID: "e386c89b-ffb6-45a3-b00e-c99c405c0303"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:40:31.047813 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.047783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e386c89b-ffb6-45a3-b00e-c99c405c0303" (UID: "e386c89b-ffb6-45a3-b00e-c99c405c0303"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:40:31.047916 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.047881    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e386c89b-ffb6-45a3-b00e-c99c405c0303" (UID: "e386c89b-ffb6-45a3-b00e-c99c405c0303"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:40:31.047968 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.047916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e386c89b-ffb6-45a3-b00e-c99c405c0303-kube-api-access-96msq" (OuterVolumeSpecName: "kube-api-access-96msq") pod "e386c89b-ffb6-45a3-b00e-c99c405c0303" (UID: "e386c89b-ffb6-45a3-b00e-c99c405c0303"). InnerVolumeSpecName "kube-api-access-96msq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:40:31.146897 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.146867    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.146897 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.146892    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96msq\" (UniqueName: \"kubernetes.io/projected/e386c89b-ffb6-45a3-b00e-c99c405c0303-kube-api-access-96msq\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.146897 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.146902    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e386c89b-ffb6-45a3-b00e-c99c405c0303-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.147108 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.146911    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e386c89b-ffb6-45a3-b00e-c99c405c0303-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.445004 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.444968    2579 generic.go:356] "Generic (PLEG): container finished" podID="e386c89b-ffb6-45a3-b00e-c99c405c0303" containerID="c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40" exitCode=0
Sep 15 09:40:31.445190 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.445020    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"
Sep 15 09:40:31.445190 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.445054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f" event={"ID":"e386c89b-ffb6-45a3-b00e-c99c405c0303","Type":"ContainerDied","Data":"c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40"}
Sep 15 09:40:31.445190 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.445116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f" event={"ID":"e386c89b-ffb6-45a3-b00e-c99c405c0303","Type":"ContainerDied","Data":"44763c295abc97a30273a9a28eac674b44b38deadbca21bce5bc7f66edc6eb79"}
Sep 15 09:40:31.445190 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.445138    2579 scope.go:117] "RemoveContainer" containerID="c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40"
Sep 15 09:40:31.455380 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.455354    2579 scope.go:117] "RemoveContainer" containerID="c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40"
Sep 15 09:40:31.455672 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:40:31.455652    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40\": container with ID starting with c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40 not found: ID does not exist" containerID="c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40"
Sep 15 09:40:31.455749 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.455681    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40"} err="failed to get container status \"c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40\": rpc error: code = NotFound desc = could not find container \"c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40\": container with ID starting with c6e2053ea407c5f020fce232fa6b02f60878497ee2bfcfc57f8f2d92a59d8d40 not found: ID does not exist"
Sep 15 09:40:31.467762 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.467738    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"]
Sep 15 09:40:31.471644 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:31.471620    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c9d6bb-59cb6ff4bd-xxb7f"]
Sep 15 09:40:32.675029 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:32.674990    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e386c89b-ffb6-45a3-b00e-c99c405c0303" path="/var/lib/kubelet/pods/e386c89b-ffb6-45a3-b00e-c99c405c0303/volumes"
Sep 15 09:40:53.344911 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.344874    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"]
Sep 15 09:40:53.345285 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.345128    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn" podUID="90829547-1aad-4b4b-96f2-59ec4fc2be37" containerName="maas-api" containerID="cri-o://40126d43f70c938546f136d7fa6eede5ad882885b828213fa4a1211346dc4aa5" gracePeriod=30
Sep 15 09:40:53.562813 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.562774    2579 generic.go:356] "Generic (PLEG): container finished" podID="90829547-1aad-4b4b-96f2-59ec4fc2be37" containerID="40126d43f70c938546f136d7fa6eede5ad882885b828213fa4a1211346dc4aa5" exitCode=0
Sep 15 09:40:53.562956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.562838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn" event={"ID":"90829547-1aad-4b4b-96f2-59ec4fc2be37","Type":"ContainerDied","Data":"40126d43f70c938546f136d7fa6eede5ad882885b828213fa4a1211346dc4aa5"}
Sep 15 09:40:53.594260 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.594235    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:40:53.649529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.649461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf65m\" (UniqueName: \"kubernetes.io/projected/90829547-1aad-4b4b-96f2-59ec4fc2be37-kube-api-access-gf65m\") pod \"90829547-1aad-4b4b-96f2-59ec4fc2be37\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") "
Sep 15 09:40:53.649664 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.649644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90829547-1aad-4b4b-96f2-59ec4fc2be37-openshift-service-ca\") pod \"90829547-1aad-4b4b-96f2-59ec4fc2be37\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") "
Sep 15 09:40:53.649726 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.649683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-maas-api-tls\") pod \"90829547-1aad-4b4b-96f2-59ec4fc2be37\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") "
Sep 15 09:40:53.649784 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.649737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-metrics-certs\") pod \"90829547-1aad-4b4b-96f2-59ec4fc2be37\" (UID: \"90829547-1aad-4b4b-96f2-59ec4fc2be37\") "
Sep 15 09:40:53.650038 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.650002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90829547-1aad-4b4b-96f2-59ec4fc2be37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90829547-1aad-4b4b-96f2-59ec4fc2be37" (UID: "90829547-1aad-4b4b-96f2-59ec4fc2be37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:40:53.650153 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.650112    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90829547-1aad-4b4b-96f2-59ec4fc2be37-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:40:53.651640 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.651611    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90829547-1aad-4b4b-96f2-59ec4fc2be37-kube-api-access-gf65m" (OuterVolumeSpecName: "kube-api-access-gf65m") pod "90829547-1aad-4b4b-96f2-59ec4fc2be37" (UID: "90829547-1aad-4b4b-96f2-59ec4fc2be37"). InnerVolumeSpecName "kube-api-access-gf65m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:40:53.651764 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.651661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90829547-1aad-4b4b-96f2-59ec4fc2be37" (UID: "90829547-1aad-4b4b-96f2-59ec4fc2be37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:40:53.651764 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.651754    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90829547-1aad-4b4b-96f2-59ec4fc2be37" (UID: "90829547-1aad-4b4b-96f2-59ec4fc2be37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:40:53.751598 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.751569    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf65m\" (UniqueName: \"kubernetes.io/projected/90829547-1aad-4b4b-96f2-59ec4fc2be37-kube-api-access-gf65m\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:40:53.751598 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.751596    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:40:53.751598 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:53.751605    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90829547-1aad-4b4b-96f2-59ec4fc2be37-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:40:54.568942 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:54.568910    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"
Sep 15 09:40:54.569317 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:54.568950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn" event={"ID":"90829547-1aad-4b4b-96f2-59ec4fc2be37","Type":"ContainerDied","Data":"1a1c028062df10f0fdad7d9563b30d05b2534dc936b03e46171dd5ff447c8209"}
Sep 15 09:40:54.569317 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:54.569015    2579 scope.go:117] "RemoveContainer" containerID="40126d43f70c938546f136d7fa6eede5ad882885b828213fa4a1211346dc4aa5"
Sep 15 09:40:54.594721 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:54.594692    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"]
Sep 15 09:40:54.596605 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:54.596585    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e029b0-69cbf77df-8lrmn"]
Sep 15 09:40:54.676270 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:54.676230    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90829547-1aad-4b4b-96f2-59ec4fc2be37" path="/var/lib/kubelet/pods/90829547-1aad-4b4b-96f2-59ec4fc2be37/volumes"
Sep 15 09:40:56.100509 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:40:56.100426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:41:11.133230 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.133193    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"]
Sep 15 09:41:11.133840 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.133461    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9" podUID="431c1687-f5e9-4a67-9d00-918ed8813622" containerName="maas-api" containerID="cri-o://2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424" gracePeriod=30
Sep 15 09:41:11.387997 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.387934    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:41:11.538629 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.538597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431c1687-f5e9-4a67-9d00-918ed8813622-openshift-service-ca\") pod \"431c1687-f5e9-4a67-9d00-918ed8813622\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") "
Sep 15 09:41:11.538789 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.538647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-metrics-certs\") pod \"431c1687-f5e9-4a67-9d00-918ed8813622\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") "
Sep 15 09:41:11.538789 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.538732    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-maas-api-tls\") pod \"431c1687-f5e9-4a67-9d00-918ed8813622\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") "
Sep 15 09:41:11.538789 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.538758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtlp7\" (UniqueName: \"kubernetes.io/projected/431c1687-f5e9-4a67-9d00-918ed8813622-kube-api-access-wtlp7\") pod \"431c1687-f5e9-4a67-9d00-918ed8813622\" (UID: \"431c1687-f5e9-4a67-9d00-918ed8813622\") "
Sep 15 09:41:11.539028 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.539002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431c1687-f5e9-4a67-9d00-918ed8813622-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "431c1687-f5e9-4a67-9d00-918ed8813622" (UID: "431c1687-f5e9-4a67-9d00-918ed8813622"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:41:11.540845 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.540817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "431c1687-f5e9-4a67-9d00-918ed8813622" (UID: "431c1687-f5e9-4a67-9d00-918ed8813622"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:41:11.540961 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.540861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431c1687-f5e9-4a67-9d00-918ed8813622-kube-api-access-wtlp7" (OuterVolumeSpecName: "kube-api-access-wtlp7") pod "431c1687-f5e9-4a67-9d00-918ed8813622" (UID: "431c1687-f5e9-4a67-9d00-918ed8813622"). InnerVolumeSpecName "kube-api-access-wtlp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:41:11.540961 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.540903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "431c1687-f5e9-4a67-9d00-918ed8813622" (UID: "431c1687-f5e9-4a67-9d00-918ed8813622"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:41:11.640381 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.640306    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:11.640381 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.640329    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtlp7\" (UniqueName: \"kubernetes.io/projected/431c1687-f5e9-4a67-9d00-918ed8813622-kube-api-access-wtlp7\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:11.640381 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.640350    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/431c1687-f5e9-4a67-9d00-918ed8813622-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:11.640381 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.640359    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/431c1687-f5e9-4a67-9d00-918ed8813622-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:11.665222 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.665197    2579 generic.go:356] "Generic (PLEG): container finished" podID="431c1687-f5e9-4a67-9d00-918ed8813622" containerID="2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424" exitCode=0
Sep 15 09:41:11.665328 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.665256    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"
Sep 15 09:41:11.665328 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.665281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9" event={"ID":"431c1687-f5e9-4a67-9d00-918ed8813622","Type":"ContainerDied","Data":"2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424"}
Sep 15 09:41:11.665430 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.665364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9" event={"ID":"431c1687-f5e9-4a67-9d00-918ed8813622","Type":"ContainerDied","Data":"88329e11f9f5fe52f70c7a046496f3aecd1a571d47e3ee49e7938f44c45bed4f"}
Sep 15 09:41:11.665430 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.665398    2579 scope.go:117] "RemoveContainer" containerID="2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424"
Sep 15 09:41:11.674052 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.674037    2579 scope.go:117] "RemoveContainer" containerID="2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424"
Sep 15 09:41:11.674355 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:41:11.674317    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424\": container with ID starting with 2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424 not found: ID does not exist" containerID="2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424"
Sep 15 09:41:11.674468 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.674449    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424"} err="failed to get container status \"2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424\": rpc error: code = NotFound desc = could not find container \"2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424\": container with ID starting with 2959ac1d047c952b17cb5a46148b17afa683d0215dce732788db8b9cdaceb424 not found: ID does not exist"
Sep 15 09:41:11.687269 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.687247    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"]
Sep 15 09:41:11.689227 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:11.689208    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f514a4-9dd994d96-gqqr9"]
Sep 15 09:41:12.675056 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:12.675026    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="431c1687-f5e9-4a67-9d00-918ed8813622" path="/var/lib/kubelet/pods/431c1687-f5e9-4a67-9d00-918ed8813622/volumes"
Sep 15 09:41:22.192849 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:22.192802    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"]
Sep 15 09:41:22.255981 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:22.255946    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"]
Sep 15 09:41:29.141487 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.141455    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"]
Sep 15 09:41:29.141935 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.141694    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt" podUID="f31ec4b0-5806-4388-a725-8cccef3f2f85" containerName="maas-api" containerID="cri-o://6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637" gracePeriod=30
Sep 15 09:41:29.394975 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.394908    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:41:29.509029 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.508999    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-metrics-certs\") pod \"f31ec4b0-5806-4388-a725-8cccef3f2f85\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") "
Sep 15 09:41:29.509196 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.509052    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqzrg\" (UniqueName: \"kubernetes.io/projected/f31ec4b0-5806-4388-a725-8cccef3f2f85-kube-api-access-zqzrg\") pod \"f31ec4b0-5806-4388-a725-8cccef3f2f85\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") "
Sep 15 09:41:29.509196 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.509088    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-maas-api-tls\") pod \"f31ec4b0-5806-4388-a725-8cccef3f2f85\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") "
Sep 15 09:41:29.509196 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.509160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ec4b0-5806-4388-a725-8cccef3f2f85-openshift-service-ca\") pod \"f31ec4b0-5806-4388-a725-8cccef3f2f85\" (UID: \"f31ec4b0-5806-4388-a725-8cccef3f2f85\") "
Sep 15 09:41:29.509675 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.509643    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31ec4b0-5806-4388-a725-8cccef3f2f85-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f31ec4b0-5806-4388-a725-8cccef3f2f85" (UID: "f31ec4b0-5806-4388-a725-8cccef3f2f85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:41:29.511232 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.511195    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f31ec4b0-5806-4388-a725-8cccef3f2f85" (UID: "f31ec4b0-5806-4388-a725-8cccef3f2f85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:41:29.511362 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.511236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f31ec4b0-5806-4388-a725-8cccef3f2f85" (UID: "f31ec4b0-5806-4388-a725-8cccef3f2f85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:41:29.511362 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.511253    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31ec4b0-5806-4388-a725-8cccef3f2f85-kube-api-access-zqzrg" (OuterVolumeSpecName: "kube-api-access-zqzrg") pod "f31ec4b0-5806-4388-a725-8cccef3f2f85" (UID: "f31ec4b0-5806-4388-a725-8cccef3f2f85"). InnerVolumeSpecName "kube-api-access-zqzrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:41:29.610437 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.610405    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:29.610437 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.610432    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31ec4b0-5806-4388-a725-8cccef3f2f85-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:29.610437 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.610444    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f31ec4b0-5806-4388-a725-8cccef3f2f85-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:29.610654 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.610452    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqzrg\" (UniqueName: \"kubernetes.io/projected/f31ec4b0-5806-4388-a725-8cccef3f2f85-kube-api-access-zqzrg\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:29.762976 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.762942    2579 generic.go:356] "Generic (PLEG): container finished" podID="f31ec4b0-5806-4388-a725-8cccef3f2f85" containerID="6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637" exitCode=0
Sep 15 09:41:29.763129 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.762987    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"
Sep 15 09:41:29.763129 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.763030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt" event={"ID":"f31ec4b0-5806-4388-a725-8cccef3f2f85","Type":"ContainerDied","Data":"6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637"}
Sep 15 09:41:29.763129 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.763083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt" event={"ID":"f31ec4b0-5806-4388-a725-8cccef3f2f85","Type":"ContainerDied","Data":"4538e45e05b220f5d9e4fdde22e6e69951a06e0118de711caa49d887c5e2affc"}
Sep 15 09:41:29.763129 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.763113    2579 scope.go:117] "RemoveContainer" containerID="6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637"
Sep 15 09:41:29.773122 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.773104    2579 scope.go:117] "RemoveContainer" containerID="6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637"
Sep 15 09:41:29.773429 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:41:29.773411    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637\": container with ID starting with 6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637 not found: ID does not exist" containerID="6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637"
Sep 15 09:41:29.773494 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.773437    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637"} err="failed to get container status \"6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637\": rpc error: code = NotFound desc = could not find container \"6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637\": container with ID starting with 6b5563c3415f6f22fdbfe778f6d50be72f68ee55e361a8fec5537437797a5637 not found: ID does not exist"
Sep 15 09:41:29.786712 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.786681    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"]
Sep 15 09:41:29.790583 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:29.790561    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-161187-757b44fcdd-stkdt"]
Sep 15 09:41:30.446116 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:30.446084    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"]
Sep 15 09:41:30.446550 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:30.446316    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd" podUID="06008619-a502-4318-9a7c-78d6b2f2aafb" containerName="maas-api" containerID="cri-o://f5b618b5d2af211f04a78d343da02022730dc1611f327e437a65dbfcff6beca8" gracePeriod=30
Sep 15 09:41:30.675948 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:30.675916    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f31ec4b0-5806-4388-a725-8cccef3f2f85" path="/var/lib/kubelet/pods/f31ec4b0-5806-4388-a725-8cccef3f2f85/volumes"
Sep 15 09:41:30.771194 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:30.771168    2579 generic.go:356] "Generic (PLEG): container finished" podID="06008619-a502-4318-9a7c-78d6b2f2aafb" containerID="f5b618b5d2af211f04a78d343da02022730dc1611f327e437a65dbfcff6beca8" exitCode=0
Sep 15 09:41:30.771309 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:30.771244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd" event={"ID":"06008619-a502-4318-9a7c-78d6b2f2aafb","Type":"ContainerDied","Data":"f5b618b5d2af211f04a78d343da02022730dc1611f327e437a65dbfcff6beca8"}
Sep 15 09:41:30.902040 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:30.902017    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:41:31.024928 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.024840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06008619-a502-4318-9a7c-78d6b2f2aafb-openshift-service-ca\") pod \"06008619-a502-4318-9a7c-78d6b2f2aafb\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") "
Sep 15 09:41:31.025101 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.024947    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-metrics-certs\") pod \"06008619-a502-4318-9a7c-78d6b2f2aafb\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") "
Sep 15 09:41:31.025101 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.025022    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9jh9\" (UniqueName: \"kubernetes.io/projected/06008619-a502-4318-9a7c-78d6b2f2aafb-kube-api-access-s9jh9\") pod \"06008619-a502-4318-9a7c-78d6b2f2aafb\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") "
Sep 15 09:41:31.025236 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.025126    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-maas-api-tls\") pod \"06008619-a502-4318-9a7c-78d6b2f2aafb\" (UID: \"06008619-a502-4318-9a7c-78d6b2f2aafb\") "
Sep 15 09:41:31.025236 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.025184    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06008619-a502-4318-9a7c-78d6b2f2aafb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06008619-a502-4318-9a7c-78d6b2f2aafb" (UID: "06008619-a502-4318-9a7c-78d6b2f2aafb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:41:31.025524 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.025498    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06008619-a502-4318-9a7c-78d6b2f2aafb-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:31.027080 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.027055    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06008619-a502-4318-9a7c-78d6b2f2aafb" (UID: "06008619-a502-4318-9a7c-78d6b2f2aafb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:41:31.027180 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.027100    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "06008619-a502-4318-9a7c-78d6b2f2aafb" (UID: "06008619-a502-4318-9a7c-78d6b2f2aafb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:41:31.027180 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.027121    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06008619-a502-4318-9a7c-78d6b2f2aafb-kube-api-access-s9jh9" (OuterVolumeSpecName: "kube-api-access-s9jh9") pod "06008619-a502-4318-9a7c-78d6b2f2aafb" (UID: "06008619-a502-4318-9a7c-78d6b2f2aafb"). InnerVolumeSpecName "kube-api-access-s9jh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:41:31.126406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.126373    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jh9\" (UniqueName: \"kubernetes.io/projected/06008619-a502-4318-9a7c-78d6b2f2aafb-kube-api-access-s9jh9\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:31.126406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.126402    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:31.126406 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.126411    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06008619-a502-4318-9a7c-78d6b2f2aafb-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:31.599588 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.599549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:41:31.778489 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.778452    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"
Sep 15 09:41:31.778489 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.778472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd" event={"ID":"06008619-a502-4318-9a7c-78d6b2f2aafb","Type":"ContainerDied","Data":"bc6ad7884027b870800ed87f21a9a433187f1800f9ecfe418b3d3d65ea0d2e49"}
Sep 15 09:41:31.778747 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.778542    2579 scope.go:117] "RemoveContainer" containerID="f5b618b5d2af211f04a78d343da02022730dc1611f327e437a65dbfcff6beca8"
Sep 15 09:41:31.800464 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.800435    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"]
Sep 15 09:41:31.804740 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:31.804715    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e76bac-675dd984b-9b4nd"]
Sep 15 09:41:32.675391 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:32.675360    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06008619-a502-4318-9a7c-78d6b2f2aafb" path="/var/lib/kubelet/pods/06008619-a502-4318-9a7c-78d6b2f2aafb/volumes"
Sep 15 09:41:37.212215 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.212169    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq" podUID="ea5414c4-2153-41a8-ba76-517bf3731bf1" containerName="main" containerID="cri-o://e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b" gracePeriod=45
Sep 15 09:41:37.275049 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.275006    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc" podUID="595eaf3a-aaaf-4762-870e-0a70d4838636" containerName="main" containerID="cri-o://06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e" gracePeriod=45
Sep 15 09:41:37.543864 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.543829    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:41:37.576555 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.576518    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:41:37.691211 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wszzb\" (UniqueName: \"kubernetes.io/projected/595eaf3a-aaaf-4762-870e-0a70d4838636-kube-api-access-wszzb\") pod \"595eaf3a-aaaf-4762-870e-0a70d4838636\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") "
Sep 15 09:41:37.691442 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-tmp-dir\") pod \"ea5414c4-2153-41a8-ba76-517bf3731bf1\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") "
Sep 15 09:41:37.691442 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-tmp-dir\") pod \"595eaf3a-aaaf-4762-870e-0a70d4838636\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") "
Sep 15 09:41:37.691442 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-model-cache\") pod \"ea5414c4-2153-41a8-ba76-517bf3731bf1\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") "
Sep 15 09:41:37.691442 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-dshm\") pod \"ea5414c4-2153-41a8-ba76-517bf3731bf1\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") "
Sep 15 09:41:37.691442 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691443    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5414c4-2153-41a8-ba76-517bf3731bf1-tls-certs\") pod \"ea5414c4-2153-41a8-ba76-517bf3731bf1\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") "
Sep 15 09:41:37.691779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691469    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-model-cache\") pod \"595eaf3a-aaaf-4762-870e-0a70d4838636\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") "
Sep 15 09:41:37.691779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595eaf3a-aaaf-4762-870e-0a70d4838636-tls-certs\") pod \"595eaf3a-aaaf-4762-870e-0a70d4838636\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") "
Sep 15 09:41:37.691779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691520    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ssr5\" (UniqueName: \"kubernetes.io/projected/ea5414c4-2153-41a8-ba76-517bf3731bf1-kube-api-access-2ssr5\") pod \"ea5414c4-2153-41a8-ba76-517bf3731bf1\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") "
Sep 15 09:41:37.691779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691541    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea5414c4-2153-41a8-ba76-517bf3731bf1" (UID: "ea5414c4-2153-41a8-ba76-517bf3731bf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:37.691779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691579    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-dshm\") pod \"595eaf3a-aaaf-4762-870e-0a70d4838636\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") "
Sep 15 09:41:37.691779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691632    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-home\") pod \"595eaf3a-aaaf-4762-870e-0a70d4838636\" (UID: \"595eaf3a-aaaf-4762-870e-0a70d4838636\") "
Sep 15 09:41:37.691779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691633    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "595eaf3a-aaaf-4762-870e-0a70d4838636" (UID: "595eaf3a-aaaf-4762-870e-0a70d4838636"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:37.691779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea5414c4-2153-41a8-ba76-517bf3731bf1" (UID: "ea5414c4-2153-41a8-ba76-517bf3731bf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:37.691779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-home\") pod \"ea5414c4-2153-41a8-ba76-517bf3731bf1\" (UID: \"ea5414c4-2153-41a8-ba76-517bf3731bf1\") "
Sep 15 09:41:37.692231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-model-cache" (OuterVolumeSpecName: "model-cache") pod "595eaf3a-aaaf-4762-870e-0a70d4838636" (UID: "595eaf3a-aaaf-4762-870e-0a70d4838636"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:37.692231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.691849    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-home" (OuterVolumeSpecName: "home") pod "595eaf3a-aaaf-4762-870e-0a70d4838636" (UID: "595eaf3a-aaaf-4762-870e-0a70d4838636"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:37.692231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.692060    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-home" (OuterVolumeSpecName: "home") pod "ea5414c4-2153-41a8-ba76-517bf3731bf1" (UID: "ea5414c4-2153-41a8-ba76-517bf3731bf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:37.692231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.692124    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.692231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.692142    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.692231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.692155    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.692231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.692168    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.692231 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.692180    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.694321 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.694285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-dshm" (OuterVolumeSpecName: "dshm") pod "ea5414c4-2153-41a8-ba76-517bf3731bf1" (UID: "ea5414c4-2153-41a8-ba76-517bf3731bf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:37.694321 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.694316    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5414c4-2153-41a8-ba76-517bf3731bf1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea5414c4-2153-41a8-ba76-517bf3731bf1" (UID: "ea5414c4-2153-41a8-ba76-517bf3731bf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:41:37.694500 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.694389    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-dshm" (OuterVolumeSpecName: "dshm") pod "595eaf3a-aaaf-4762-870e-0a70d4838636" (UID: "595eaf3a-aaaf-4762-870e-0a70d4838636"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:37.694500 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.694398    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595eaf3a-aaaf-4762-870e-0a70d4838636-kube-api-access-wszzb" (OuterVolumeSpecName: "kube-api-access-wszzb") pod "595eaf3a-aaaf-4762-870e-0a70d4838636" (UID: "595eaf3a-aaaf-4762-870e-0a70d4838636"). InnerVolumeSpecName "kube-api-access-wszzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:41:37.694500 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.694389    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595eaf3a-aaaf-4762-870e-0a70d4838636-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "595eaf3a-aaaf-4762-870e-0a70d4838636" (UID: "595eaf3a-aaaf-4762-870e-0a70d4838636"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:41:37.694500 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.694438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5414c4-2153-41a8-ba76-517bf3731bf1-kube-api-access-2ssr5" (OuterVolumeSpecName: "kube-api-access-2ssr5") pod "ea5414c4-2153-41a8-ba76-517bf3731bf1" (UID: "ea5414c4-2153-41a8-ba76-517bf3731bf1"). InnerVolumeSpecName "kube-api-access-2ssr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:41:37.793816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.793725    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.793816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.793760    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5414c4-2153-41a8-ba76-517bf3731bf1-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.793816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.793770    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595eaf3a-aaaf-4762-870e-0a70d4838636-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.793816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.793778    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ssr5\" (UniqueName: \"kubernetes.io/projected/ea5414c4-2153-41a8-ba76-517bf3731bf1-kube-api-access-2ssr5\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.793816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.793787    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595eaf3a-aaaf-4762-870e-0a70d4838636-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.793816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.793794    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5414c4-2153-41a8-ba76-517bf3731bf1-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.793816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.793802    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wszzb\" (UniqueName: \"kubernetes.io/projected/595eaf3a-aaaf-4762-870e-0a70d4838636-kube-api-access-wszzb\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:41:37.813548 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.813507    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea5414c4-2153-41a8-ba76-517bf3731bf1" containerID="e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b" exitCode=0
Sep 15 09:41:37.813743 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.813554    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"
Sep 15 09:41:37.813743 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.813566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq" event={"ID":"ea5414c4-2153-41a8-ba76-517bf3731bf1","Type":"ContainerDied","Data":"e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b"}
Sep 15 09:41:37.813743 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.813610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq" event={"ID":"ea5414c4-2153-41a8-ba76-517bf3731bf1","Type":"ContainerDied","Data":"e83302ffd89d2698885ee42fc583aeb8aeaf98422e0802e415aa4066cac6c998"}
Sep 15 09:41:37.813743 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.813628    2579 scope.go:117] "RemoveContainer" containerID="e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b"
Sep 15 09:41:37.815300 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.815269    2579 generic.go:356] "Generic (PLEG): container finished" podID="595eaf3a-aaaf-4762-870e-0a70d4838636" containerID="06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e" exitCode=0
Sep 15 09:41:37.815444 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.815312    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"
Sep 15 09:41:37.815444 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.815370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc" event={"ID":"595eaf3a-aaaf-4762-870e-0a70d4838636","Type":"ContainerDied","Data":"06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e"}
Sep 15 09:41:37.815444 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.815409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc" event={"ID":"595eaf3a-aaaf-4762-870e-0a70d4838636","Type":"ContainerDied","Data":"b4e67dda3263da52ee83e05e2e170edffda00c21cd6d0aac9a6afe97b0c33619"}
Sep 15 09:41:37.823838 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.823815    2579 scope.go:117] "RemoveContainer" containerID="e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b"
Sep 15 09:41:37.824227 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:41:37.824202    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b\": container with ID starting with e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b not found: ID does not exist" containerID="e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b"
Sep 15 09:41:37.824289 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.824242    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b"} err="failed to get container status \"e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b\": rpc error: code = NotFound desc = could not find container \"e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b\": container with ID starting with e3e1fc5bdf3a197233fda2b109b1ca83200f021bff023a7141a0212405645f2b not found: ID does not exist"
Sep 15 09:41:37.824289 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.824261    2579 scope.go:117] "RemoveContainer" containerID="06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e"
Sep 15 09:41:37.833671 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.833650    2579 scope.go:117] "RemoveContainer" containerID="06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e"
Sep 15 09:41:37.834013 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:41:37.833991    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e\": container with ID starting with 06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e not found: ID does not exist" containerID="06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e"
Sep 15 09:41:37.834091 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.834025    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e"} err="failed to get container status \"06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e\": rpc error: code = NotFound desc = could not find container \"06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e\": container with ID starting with 06be7551c7e12b8a67f59f62fb5a32f91c43bdefbe0319cabb28336f3b299f4e not found: ID does not exist"
Sep 15 09:41:37.835042 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.835021    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"]
Sep 15 09:41:37.840922 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.840897    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/distinct-w0-e76bac-kserve-745989754c-jn5dq"]
Sep 15 09:41:37.850695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.850650    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"]
Sep 15 09:41:37.853678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:37.853649    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e76bac/simulator-w0-e76bac-kserve-7df47b4b57-rg5lc"]
Sep 15 09:41:38.676618 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:38.676586    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="595eaf3a-aaaf-4762-870e-0a70d4838636" path="/var/lib/kubelet/pods/595eaf3a-aaaf-4762-870e-0a70d4838636/volumes"
Sep 15 09:41:38.677008 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:41:38.676994    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea5414c4-2153-41a8-ba76-517bf3731bf1" path="/var/lib/kubelet/pods/ea5414c4-2153-41a8-ba76-517bf3731bf1/volumes"
Sep 15 09:42:01.090554 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:01.090521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:42:22.498944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:22.498869    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:42:22.935736 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:22.935701    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"]
Sep 15 09:42:22.945174 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:22.945147    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"]
Sep 15 09:42:22.951157 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:22.951128    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"]
Sep 15 09:42:33.564164 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:33.564123    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"]
Sep 15 09:42:33.564797 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:33.564431    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8" podUID="52397807-e1d5-4274-95f7-bf88ddd0d345" containerName="maas-api" containerID="cri-o://bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25" gracePeriod=30
Sep 15 09:42:34.021755 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.021725    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:42:34.106467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.106318    2579 generic.go:356] "Generic (PLEG): container finished" podID="52397807-e1d5-4274-95f7-bf88ddd0d345" containerID="bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25" exitCode=0
Sep 15 09:42:34.106467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.106397    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"
Sep 15 09:42:34.106467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.106404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8" event={"ID":"52397807-e1d5-4274-95f7-bf88ddd0d345","Type":"ContainerDied","Data":"bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25"}
Sep 15 09:42:34.106746 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.106474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8" event={"ID":"52397807-e1d5-4274-95f7-bf88ddd0d345","Type":"ContainerDied","Data":"7a20957fab462a4bc56388f703a5cc3798ff37de99669f92d2d660ad15eb226e"}
Sep 15 09:42:34.106746 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.106502    2579 scope.go:117] "RemoveContainer" containerID="bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25"
Sep 15 09:42:34.116155 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.116133    2579 scope.go:117] "RemoveContainer" containerID="bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25"
Sep 15 09:42:34.116588 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:42:34.116567    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25\": container with ID starting with bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25 not found: ID does not exist" containerID="bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25"
Sep 15 09:42:34.116695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.116594    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25"} err="failed to get container status \"bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25\": rpc error: code = NotFound desc = could not find container \"bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25\": container with ID starting with bb3dec7e776070653ad2b4aa8e8865195dd367b76b2f7e5a2648243ec3a81b25 not found: ID does not exist"
Sep 15 09:42:34.131584 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.131543    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-maas-api-tls\") pod \"52397807-e1d5-4274-95f7-bf88ddd0d345\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") "
Sep 15 09:42:34.131584 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.131592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-metrics-certs\") pod \"52397807-e1d5-4274-95f7-bf88ddd0d345\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") "
Sep 15 09:42:34.131781 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.131683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52397807-e1d5-4274-95f7-bf88ddd0d345-openshift-service-ca\") pod \"52397807-e1d5-4274-95f7-bf88ddd0d345\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") "
Sep 15 09:42:34.131781 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.131755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59dbc\" (UniqueName: \"kubernetes.io/projected/52397807-e1d5-4274-95f7-bf88ddd0d345-kube-api-access-59dbc\") pod \"52397807-e1d5-4274-95f7-bf88ddd0d345\" (UID: \"52397807-e1d5-4274-95f7-bf88ddd0d345\") "
Sep 15 09:42:34.132063 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.132039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52397807-e1d5-4274-95f7-bf88ddd0d345-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52397807-e1d5-4274-95f7-bf88ddd0d345" (UID: "52397807-e1d5-4274-95f7-bf88ddd0d345"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:42:34.133880 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.133851    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52397807-e1d5-4274-95f7-bf88ddd0d345" (UID: "52397807-e1d5-4274-95f7-bf88ddd0d345"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:42:34.134029 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.133887    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52397807-e1d5-4274-95f7-bf88ddd0d345" (UID: "52397807-e1d5-4274-95f7-bf88ddd0d345"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:42:34.134029 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.133922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52397807-e1d5-4274-95f7-bf88ddd0d345-kube-api-access-59dbc" (OuterVolumeSpecName: "kube-api-access-59dbc") pod "52397807-e1d5-4274-95f7-bf88ddd0d345" (UID: "52397807-e1d5-4274-95f7-bf88ddd0d345"). InnerVolumeSpecName "kube-api-access-59dbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:42:34.233140 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.233101    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52397807-e1d5-4274-95f7-bf88ddd0d345-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:34.233140 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.233139    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59dbc\" (UniqueName: \"kubernetes.io/projected/52397807-e1d5-4274-95f7-bf88ddd0d345-kube-api-access-59dbc\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:34.233140 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.233155    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:34.233482 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.233167    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52397807-e1d5-4274-95f7-bf88ddd0d345-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:34.427586 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.427545    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"]
Sep 15 09:42:34.431323 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.431288    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c183a5-6b88c9bf9-64jc8"]
Sep 15 09:42:34.676229 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:34.676148    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52397807-e1d5-4274-95f7-bf88ddd0d345" path="/var/lib/kubelet/pods/52397807-e1d5-4274-95f7-bf88ddd0d345/volumes"
Sep 15 09:42:37.955458 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:37.955411    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr" podUID="a58994ca-7521-488c-8e4a-09d603fc9968" containerName="main" containerID="cri-o://6b25a24691f5e29fb00e973152bd3588f587a727080323ba060065061074f6ef" gracePeriod=45
Sep 15 09:42:37.974886 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:37.974742    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr" podUID="94b9b9e8-2c21-4567-8269-365fa836fb82" containerName="main" containerID="cri-o://9f5fc07187939af66a3972033195045bcc019407764fa505151f450d7dc60999" gracePeriod=45
Sep 15 09:42:37.974886 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:37.974742    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9" podUID="7b750eca-e525-4275-8978-8c459e9cef2b" containerName="main" containerID="cri-o://b31b9148567de1664f78afea9c3683b76ba0dbae2637b71c5c36d06ab66ea101" gracePeriod=45
Sep 15 09:42:38.130296 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.130243    2579 generic.go:356] "Generic (PLEG): container finished" podID="a58994ca-7521-488c-8e4a-09d603fc9968" containerID="6b25a24691f5e29fb00e973152bd3588f587a727080323ba060065061074f6ef" exitCode=0
Sep 15 09:42:38.130493 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.130329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr" event={"ID":"a58994ca-7521-488c-8e4a-09d603fc9968","Type":"ContainerDied","Data":"6b25a24691f5e29fb00e973152bd3588f587a727080323ba060065061074f6ef"}
Sep 15 09:42:38.132254 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.132227    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b750eca-e525-4275-8978-8c459e9cef2b" containerID="b31b9148567de1664f78afea9c3683b76ba0dbae2637b71c5c36d06ab66ea101" exitCode=0
Sep 15 09:42:38.132410 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.132309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9" event={"ID":"7b750eca-e525-4275-8978-8c459e9cef2b","Type":"ContainerDied","Data":"b31b9148567de1664f78afea9c3683b76ba0dbae2637b71c5c36d06ab66ea101"}
Sep 15 09:42:38.134203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.134181    2579 generic.go:356] "Generic (PLEG): container finished" podID="94b9b9e8-2c21-4567-8269-365fa836fb82" containerID="9f5fc07187939af66a3972033195045bcc019407764fa505151f450d7dc60999" exitCode=0
Sep 15 09:42:38.134330 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.134230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr" event={"ID":"94b9b9e8-2c21-4567-8269-365fa836fb82","Type":"ContainerDied","Data":"9f5fc07187939af66a3972033195045bcc019407764fa505151f450d7dc60999"}
Sep 15 09:42:38.263412 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.263375    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:42:38.266649 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.266625    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:42:38.270943 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.270920    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:42:38.374678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-dshm\") pod \"7b750eca-e525-4275-8978-8c459e9cef2b\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") "
Sep 15 09:42:38.374829 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-model-cache\") pod \"7b750eca-e525-4275-8978-8c459e9cef2b\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") "
Sep 15 09:42:38.374829 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-tmp-dir\") pod \"94b9b9e8-2c21-4567-8269-365fa836fb82\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") "
Sep 15 09:42:38.374829 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374740    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-tmp-dir\") pod \"a58994ca-7521-488c-8e4a-09d603fc9968\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") "
Sep 15 09:42:38.374829 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374786    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-home\") pod \"7b750eca-e525-4275-8978-8c459e9cef2b\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") "
Sep 15 09:42:38.374829 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374817    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsxk7\" (UniqueName: \"kubernetes.io/projected/a58994ca-7521-488c-8e4a-09d603fc9968-kube-api-access-gsxk7\") pod \"a58994ca-7521-488c-8e4a-09d603fc9968\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") "
Sep 15 09:42:38.375097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-model-cache\") pod \"a58994ca-7521-488c-8e4a-09d603fc9968\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") "
Sep 15 09:42:38.375097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-model-cache\") pod \"94b9b9e8-2c21-4567-8269-365fa836fb82\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") "
Sep 15 09:42:38.375097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374928    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-dshm\") pod \"94b9b9e8-2c21-4567-8269-365fa836fb82\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") "
Sep 15 09:42:38.375097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374967    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94b9b9e8-2c21-4567-8269-365fa836fb82" (UID: "94b9b9e8-2c21-4567-8269-365fa836fb82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.375097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374975    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pdsc\" (UniqueName: \"kubernetes.io/projected/7b750eca-e525-4275-8978-8c459e9cef2b-kube-api-access-4pdsc\") pod \"7b750eca-e525-4275-8978-8c459e9cef2b\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") "
Sep 15 09:42:38.375097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.374973    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b750eca-e525-4275-8978-8c459e9cef2b" (UID: "7b750eca-e525-4275-8978-8c459e9cef2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.375097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375021    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b750eca-e525-4275-8978-8c459e9cef2b-tls-certs\") pod \"7b750eca-e525-4275-8978-8c459e9cef2b\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") "
Sep 15 09:42:38.375097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375020    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a58994ca-7521-488c-8e4a-09d603fc9968" (UID: "a58994ca-7521-488c-8e4a-09d603fc9968"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.375097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375019    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-home" (OuterVolumeSpecName: "home") pod "7b750eca-e525-4275-8978-8c459e9cef2b" (UID: "7b750eca-e525-4275-8978-8c459e9cef2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.375591 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375107    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lssxh\" (UniqueName: \"kubernetes.io/projected/94b9b9e8-2c21-4567-8269-365fa836fb82-kube-api-access-lssxh\") pod \"94b9b9e8-2c21-4567-8269-365fa836fb82\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") "
Sep 15 09:42:38.375591 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375154    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-dshm\") pod \"a58994ca-7521-488c-8e4a-09d603fc9968\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") "
Sep 15 09:42:38.375591 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375183    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58994ca-7521-488c-8e4a-09d603fc9968-tls-certs\") pod \"a58994ca-7521-488c-8e4a-09d603fc9968\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") "
Sep 15 09:42:38.375591 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375209    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-tmp-dir\") pod \"7b750eca-e525-4275-8978-8c459e9cef2b\" (UID: \"7b750eca-e525-4275-8978-8c459e9cef2b\") "
Sep 15 09:42:38.375591 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375220    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-model-cache" (OuterVolumeSpecName: "model-cache") pod "a58994ca-7521-488c-8e4a-09d603fc9968" (UID: "a58994ca-7521-488c-8e4a-09d603fc9968"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.375591 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-home\") pod \"a58994ca-7521-488c-8e4a-09d603fc9968\" (UID: \"a58994ca-7521-488c-8e4a-09d603fc9968\") "
Sep 15 09:42:38.375591 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375296    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b9b9e8-2c21-4567-8269-365fa836fb82-tls-certs\") pod \"94b9b9e8-2c21-4567-8269-365fa836fb82\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") "
Sep 15 09:42:38.375591 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-home\") pod \"94b9b9e8-2c21-4567-8269-365fa836fb82\" (UID: \"94b9b9e8-2c21-4567-8269-365fa836fb82\") "
Sep 15 09:42:38.375980 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375900    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.375980 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375922    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.375980 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375935    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.375980 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375949    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.375980 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375962    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.376242 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-model-cache" (OuterVolumeSpecName: "model-cache") pod "94b9b9e8-2c21-4567-8269-365fa836fb82" (UID: "94b9b9e8-2c21-4567-8269-365fa836fb82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.376242 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.375414    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-home" (OuterVolumeSpecName: "home") pod "a58994ca-7521-488c-8e4a-09d603fc9968" (UID: "a58994ca-7521-488c-8e4a-09d603fc9968"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.376242 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.376158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-home" (OuterVolumeSpecName: "home") pod "94b9b9e8-2c21-4567-8269-365fa836fb82" (UID: "94b9b9e8-2c21-4567-8269-365fa836fb82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.377601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.377485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-dshm" (OuterVolumeSpecName: "dshm") pod "7b750eca-e525-4275-8978-8c459e9cef2b" (UID: "7b750eca-e525-4275-8978-8c459e9cef2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.377601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.377508    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b750eca-e525-4275-8978-8c459e9cef2b-kube-api-access-4pdsc" (OuterVolumeSpecName: "kube-api-access-4pdsc") pod "7b750eca-e525-4275-8978-8c459e9cef2b" (UID: "7b750eca-e525-4275-8978-8c459e9cef2b"). InnerVolumeSpecName "kube-api-access-4pdsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:42:38.377601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.377515    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b750eca-e525-4275-8978-8c459e9cef2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b750eca-e525-4275-8978-8c459e9cef2b" (UID: "7b750eca-e525-4275-8978-8c459e9cef2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:42:38.377863 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.377661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58994ca-7521-488c-8e4a-09d603fc9968-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a58994ca-7521-488c-8e4a-09d603fc9968" (UID: "a58994ca-7521-488c-8e4a-09d603fc9968"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:42:38.377863 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.377665    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-dshm" (OuterVolumeSpecName: "dshm") pod "a58994ca-7521-488c-8e4a-09d603fc9968" (UID: "a58994ca-7521-488c-8e4a-09d603fc9968"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.377863 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.377710    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b750eca-e525-4275-8978-8c459e9cef2b" (UID: "7b750eca-e525-4275-8978-8c459e9cef2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.377863 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.377800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-dshm" (OuterVolumeSpecName: "dshm") pod "94b9b9e8-2c21-4567-8269-365fa836fb82" (UID: "94b9b9e8-2c21-4567-8269-365fa836fb82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:42:38.378518 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.378491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b9b9e8-2c21-4567-8269-365fa836fb82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94b9b9e8-2c21-4567-8269-365fa836fb82" (UID: "94b9b9e8-2c21-4567-8269-365fa836fb82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:42:38.378518 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.378501    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58994ca-7521-488c-8e4a-09d603fc9968-kube-api-access-gsxk7" (OuterVolumeSpecName: "kube-api-access-gsxk7") pod "a58994ca-7521-488c-8e4a-09d603fc9968" (UID: "a58994ca-7521-488c-8e4a-09d603fc9968"). InnerVolumeSpecName "kube-api-access-gsxk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:42:38.379166 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.379148    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b9b9e8-2c21-4567-8269-365fa836fb82-kube-api-access-lssxh" (OuterVolumeSpecName: "kube-api-access-lssxh") pod "94b9b9e8-2c21-4567-8269-365fa836fb82" (UID: "94b9b9e8-2c21-4567-8269-365fa836fb82"). InnerVolumeSpecName "kube-api-access-lssxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:42:38.476744 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476720    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476744 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476741    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476751    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pdsc\" (UniqueName: \"kubernetes.io/projected/7b750eca-e525-4275-8978-8c459e9cef2b-kube-api-access-4pdsc\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476759    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b750eca-e525-4275-8978-8c459e9cef2b-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476767    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lssxh\" (UniqueName: \"kubernetes.io/projected/94b9b9e8-2c21-4567-8269-365fa836fb82-kube-api-access-lssxh\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476775    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476783    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58994ca-7521-488c-8e4a-09d603fc9968-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476792    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476799    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58994ca-7521-488c-8e4a-09d603fc9968-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476806    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b9b9e8-2c21-4567-8269-365fa836fb82-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476813    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b9b9e8-2c21-4567-8269-365fa836fb82-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476820    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b750eca-e525-4275-8978-8c459e9cef2b-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:38.476870 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:38.476827    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsxk7\" (UniqueName: \"kubernetes.io/projected/a58994ca-7521-488c-8e4a-09d603fc9968-kube-api-access-gsxk7\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:42:39.140162 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.140132    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"
Sep 15 09:42:39.140614 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.140181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr" event={"ID":"a58994ca-7521-488c-8e4a-09d603fc9968","Type":"ContainerDied","Data":"49033a9dd7afec61a98ad6dc7fb5270530a833f7162b81427757ad8dfb36869d"}
Sep 15 09:42:39.140614 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.140232    2579 scope.go:117] "RemoveContainer" containerID="6b25a24691f5e29fb00e973152bd3588f587a727080323ba060065061074f6ef"
Sep 15 09:42:39.141732 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.141674    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"
Sep 15 09:42:39.141803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.141669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9" event={"ID":"7b750eca-e525-4275-8978-8c459e9cef2b","Type":"ContainerDied","Data":"b092c3a368e704e2835d3950aa27ede21d4ecafeb12ae9dcedba99bfb2297ede"}
Sep 15 09:42:39.143283 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.143255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr" event={"ID":"94b9b9e8-2c21-4567-8269-365fa836fb82","Type":"ContainerDied","Data":"103a7d994ba60a996e356b70e6483d638fe70f0d4d633a3f9882e780189bf719"}
Sep 15 09:42:39.143447 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.143280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"
Sep 15 09:42:39.149956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.149939    2579 scope.go:117] "RemoveContainer" containerID="b31b9148567de1664f78afea9c3683b76ba0dbae2637b71c5c36d06ab66ea101"
Sep 15 09:42:39.157267 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.157252    2579 scope.go:117] "RemoveContainer" containerID="9f5fc07187939af66a3972033195045bcc019407764fa505151f450d7dc60999"
Sep 15 09:42:39.160946 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.160924    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"]
Sep 15 09:42:39.164538 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.164517    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/premium-w4-c183a5-kserve-6c7c99dc7-2q2c9"]
Sep 15 09:42:39.176697 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.176676    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"]
Sep 15 09:42:39.182315 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.182295    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/simulator-w4-c183a5-kserve-584456798d-vhllr"]
Sep 15 09:42:39.192570 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.192543    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"]
Sep 15 09:42:39.194297 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:39.194280    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c183a5/distinct-w4-c183a5-kserve-6bdc968dd8-gzxgr"]
Sep 15 09:42:40.675477 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:40.675440    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b750eca-e525-4275-8978-8c459e9cef2b" path="/var/lib/kubelet/pods/7b750eca-e525-4275-8978-8c459e9cef2b/volumes"
Sep 15 09:42:40.675862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:40.675847    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b9b9e8-2c21-4567-8269-365fa836fb82" path="/var/lib/kubelet/pods/94b9b9e8-2c21-4567-8269-365fa836fb82/volumes"
Sep 15 09:42:40.676197 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:40.676186    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58994ca-7521-488c-8e4a-09d603fc9968" path="/var/lib/kubelet/pods/a58994ca-7521-488c-8e4a-09d603fc9968/volumes"
Sep 15 09:42:44.794550 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:44.794514    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:42:56.531305 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531264    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7558-589563"]
Sep 15 09:42:56.531695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531669    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06008619-a502-4318-9a7c-78d6b2f2aafb" containerName="maas-api"
Sep 15 09:42:56.531695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531681    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="06008619-a502-4318-9a7c-78d6b2f2aafb" containerName="maas-api"
Sep 15 09:42:56.531695 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531691    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90829547-1aad-4b4b-96f2-59ec4fc2be37" containerName="maas-api"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531698    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="90829547-1aad-4b4b-96f2-59ec4fc2be37" containerName="maas-api"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531708    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58994ca-7521-488c-8e4a-09d603fc9968" containerName="main"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531713    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58994ca-7521-488c-8e4a-09d603fc9968" containerName="main"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531725    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52397807-e1d5-4274-95f7-bf88ddd0d345" containerName="maas-api"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531730    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52397807-e1d5-4274-95f7-bf88ddd0d345" containerName="maas-api"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531735    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431c1687-f5e9-4a67-9d00-918ed8813622" containerName="maas-api"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531741    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="431c1687-f5e9-4a67-9d00-918ed8813622" containerName="maas-api"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531751    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b750eca-e525-4275-8978-8c459e9cef2b" containerName="main"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531757    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b750eca-e525-4275-8978-8c459e9cef2b" containerName="main"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531763    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595eaf3a-aaaf-4762-870e-0a70d4838636" containerName="main"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531768    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="595eaf3a-aaaf-4762-870e-0a70d4838636" containerName="main"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531781    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea5414c4-2153-41a8-ba76-517bf3731bf1" containerName="main"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531787    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea5414c4-2153-41a8-ba76-517bf3731bf1" containerName="main"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531793    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f31ec4b0-5806-4388-a725-8cccef3f2f85" containerName="maas-api"
Sep 15 09:42:56.531794 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531798    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f31ec4b0-5806-4388-a725-8cccef3f2f85" containerName="maas-api"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531804    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b9b9e8-2c21-4567-8269-365fa836fb82" containerName="main"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531809    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b9b9e8-2c21-4567-8269-365fa836fb82" containerName="main"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531815    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e386c89b-ffb6-45a3-b00e-c99c405c0303" containerName="maas-api"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531820    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e386c89b-ffb6-45a3-b00e-c99c405c0303" containerName="maas-api"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b750eca-e525-4275-8978-8c459e9cef2b" containerName="main"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531899    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52397807-e1d5-4274-95f7-bf88ddd0d345" containerName="maas-api"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531905    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f31ec4b0-5806-4388-a725-8cccef3f2f85" containerName="maas-api"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531912    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="431c1687-f5e9-4a67-9d00-918ed8813622" containerName="maas-api"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531917    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="595eaf3a-aaaf-4762-870e-0a70d4838636" containerName="main"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531922    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="90829547-1aad-4b4b-96f2-59ec4fc2be37" containerName="maas-api"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531929    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58994ca-7521-488c-8e4a-09d603fc9968" containerName="main"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531934    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b9b9e8-2c21-4567-8269-365fa836fb82" containerName="main"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531939    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e386c89b-ffb6-45a3-b00e-c99c405c0303" containerName="maas-api"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531944    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="06008619-a502-4318-9a7c-78d6b2f2aafb" containerName="maas-api"
Sep 15 09:42:56.532243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.531949    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea5414c4-2153-41a8-ba76-517bf3731bf1" containerName="main"
Sep 15 09:42:56.536703 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.536685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-589563"
Sep 15 09:42:56.539215 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.539180    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-88s6f\""
Sep 15 09:42:56.539490 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.539471    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7558-589563"]
Sep 15 09:42:56.621565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.621530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drmzh\" (UniqueName: \"kubernetes.io/projected/1b50bfb4-cedc-4440-a682-4ab4264074bf-kube-api-access-drmzh\") pod \"test-curl-7558-589563\" (UID: \"1b50bfb4-cedc-4440-a682-4ab4264074bf\") " pod="opendatahub/test-curl-7558-589563"
Sep 15 09:42:56.722711 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.722669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drmzh\" (UniqueName: \"kubernetes.io/projected/1b50bfb4-cedc-4440-a682-4ab4264074bf-kube-api-access-drmzh\") pod \"test-curl-7558-589563\" (UID: \"1b50bfb4-cedc-4440-a682-4ab4264074bf\") " pod="opendatahub/test-curl-7558-589563"
Sep 15 09:42:56.731655 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.731623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drmzh\" (UniqueName: \"kubernetes.io/projected/1b50bfb4-cedc-4440-a682-4ab4264074bf-kube-api-access-drmzh\") pod \"test-curl-7558-589563\" (UID: \"1b50bfb4-cedc-4440-a682-4ab4264074bf\") " pod="opendatahub/test-curl-7558-589563"
Sep 15 09:42:56.850220 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.850133    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-88s6f\""
Sep 15 09:42:56.858871 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:56.858837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-589563"
Sep 15 09:42:57.134640 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:57.134614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7558-589563"]
Sep 15 09:42:57.136844 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:42:57.136810    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b50bfb4_cedc_4440_a682_4ab4264074bf.slice/crio-3edf08124802cf16c4e80b8f65b68e6f3d80f115f3d778ea8ed298321de75f34 WatchSource:0}: Error finding container 3edf08124802cf16c4e80b8f65b68e6f3d80f115f3d778ea8ed298321de75f34: Status 404 returned error can't find the container with id 3edf08124802cf16c4e80b8f65b68e6f3d80f115f3d778ea8ed298321de75f34
Sep 15 09:42:57.239116 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:57.239077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7558-589563" event={"ID":"1b50bfb4-cedc-4440-a682-4ab4264074bf","Type":"ContainerStarted","Data":"fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed"}
Sep 15 09:42:57.239116 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:57.239123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7558-589563" event={"ID":"1b50bfb4-cedc-4440-a682-4ab4264074bf","Type":"ContainerStarted","Data":"3edf08124802cf16c4e80b8f65b68e6f3d80f115f3d778ea8ed298321de75f34"}
Sep 15 09:42:57.254125 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:57.254067    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7558-589563" podStartSLOduration=1.254049102 podStartE2EDuration="1.254049102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:42:57.252674857 +0000 UTC m=+1741.082466761" watchObservedRunningTime="2026-09-15 09:42:57.254049102 +0000 UTC m=+1741.083841069"
Sep 15 09:42:57.844037 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:57.843935    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7558-589563"]
Sep 15 09:42:57.848252 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:57.848220    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7558-589563"]
Sep 15 09:42:58.244663 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:42:58.244622    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7558-589563" container="test-curl-7558-589563"
Sep 15 09:42:59.190259 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:59.190223    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:42:59.254121 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:42:59.254064    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7558-589563" podUID="1b50bfb4-cedc-4440-a682-4ab4264074bf" containerName="test-curl-7558-589563" containerID="cri-o://fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed" gracePeriod=2
Sep 15 09:43:00.982639 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:00.982604    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"]
Sep 15 09:43:01.460243 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:01.460210    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-589563"
Sep 15 09:43:01.574685 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:01.574581    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drmzh\" (UniqueName: \"kubernetes.io/projected/1b50bfb4-cedc-4440-a682-4ab4264074bf-kube-api-access-drmzh\") pod \"1b50bfb4-cedc-4440-a682-4ab4264074bf\" (UID: \"1b50bfb4-cedc-4440-a682-4ab4264074bf\") "
Sep 15 09:43:01.576816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:01.576785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b50bfb4-cedc-4440-a682-4ab4264074bf-kube-api-access-drmzh" (OuterVolumeSpecName: "kube-api-access-drmzh") pod "1b50bfb4-cedc-4440-a682-4ab4264074bf" (UID: "1b50bfb4-cedc-4440-a682-4ab4264074bf"). InnerVolumeSpecName "kube-api-access-drmzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:43:01.675825 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:01.675789    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drmzh\" (UniqueName: \"kubernetes.io/projected/1b50bfb4-cedc-4440-a682-4ab4264074bf-kube-api-access-drmzh\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:43:02.270010 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:02.269978    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b50bfb4-cedc-4440-a682-4ab4264074bf" containerID="fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed" exitCode=137
Sep 15 09:43:02.270502 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:02.270017    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-589563"
Sep 15 09:43:02.270502 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:02.270078    2579 scope.go:117] "RemoveContainer" containerID="fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed"
Sep 15 09:43:02.278567 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:02.278550    2579 scope.go:117] "RemoveContainer" containerID="fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed"
Sep 15 09:43:02.278846 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:43:02.278826    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed\": container with ID starting with fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed not found: ID does not exist" containerID="fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed"
Sep 15 09:43:02.278894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:02.278854    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed"} err="failed to get container status \"fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed\": rpc error: code = NotFound desc = could not find container \"fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed\": container with ID starting with fa8c54a22b9b75614309618a5882f92c45717cf9b70ccbbbdd86c765a985a6ed not found: ID does not exist"
Sep 15 09:43:02.279820 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:43:02.279799    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7558-589563\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="1b50bfb4-cedc-4440-a682-4ab4264074bf" pod="opendatahub/test-curl-7558-589563"
Sep 15 09:43:02.674720 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:02.674691    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b50bfb4-cedc-4440-a682-4ab4264074bf" path="/var/lib/kubelet/pods/1b50bfb4-cedc-4440-a682-4ab4264074bf/volumes"
Sep 15 09:43:09.607987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:09.607951    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:43:10.703830 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.703800    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"]
Sep 15 09:43:10.704286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.704159    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b50bfb4-cedc-4440-a682-4ab4264074bf" containerName="test-curl-7558-589563"
Sep 15 09:43:10.704286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.704171    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b50bfb4-cedc-4440-a682-4ab4264074bf" containerName="test-curl-7558-589563"
Sep 15 09:43:10.704286 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.704241    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b50bfb4-cedc-4440-a682-4ab4264074bf" containerName="test-curl-7558-589563"
Sep 15 09:43:10.708783 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.708763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.711105 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.711086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-864b9a\"/\"sub-test-model-864b9a-kserve-self-signed-certs\""
Sep 15 09:43:10.714492 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.714473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"]
Sep 15 09:43:10.861734 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.861702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-tmp-dir\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.861901 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.861754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-home\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.861901 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.861848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf7611a-de18-4b29-9739-68f74dcd611d-tls-certs\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.861901 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.861882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-dshm\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.862033 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.861927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxpf\" (UniqueName: \"kubernetes.io/projected/1bf7611a-de18-4b29-9739-68f74dcd611d-kube-api-access-smxpf\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.862033 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.861950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-model-cache\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.963067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.963000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-home\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.963067 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.963059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf7611a-de18-4b29-9739-68f74dcd611d-tls-certs\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.963265 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.963078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-dshm\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.963265 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.963106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxpf\" (UniqueName: \"kubernetes.io/projected/1bf7611a-de18-4b29-9739-68f74dcd611d-kube-api-access-smxpf\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.963265 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.963122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-model-cache\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.963265 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.963150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-tmp-dir\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.963507 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.963469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-home\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.963552 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.963512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-tmp-dir\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.963587 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.963570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-model-cache\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.965440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.965423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-dshm\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.965531 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.965515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf7611a-de18-4b29-9739-68f74dcd611d-tls-certs\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:10.970565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:10.970542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxpf\" (UniqueName: \"kubernetes.io/projected/1bf7611a-de18-4b29-9739-68f74dcd611d-kube-api-access-smxpf\") pod \"sub-test-model-864b9a-kserve-68867c44c-c4mtc\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:11.018440 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:11.018422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:11.291291 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:11.291266    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"]
Sep 15 09:43:11.293631 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:43:11.293606    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bf7611a_de18_4b29_9739_68f74dcd611d.slice/crio-26036769dd9f9c82ba00927dc3bc41345c4fa2d11c63c51e173f9b621b11b695 WatchSource:0}: Error finding container 26036769dd9f9c82ba00927dc3bc41345c4fa2d11c63c51e173f9b621b11b695: Status 404 returned error can't find the container with id 26036769dd9f9c82ba00927dc3bc41345c4fa2d11c63c51e173f9b621b11b695
Sep 15 09:43:11.321329 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:11.321304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc" event={"ID":"1bf7611a-de18-4b29-9739-68f74dcd611d","Type":"ContainerStarted","Data":"26036769dd9f9c82ba00927dc3bc41345c4fa2d11c63c51e173f9b621b11b695"}
Sep 15 09:43:12.327883 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:12.327832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc" event={"ID":"1bf7611a-de18-4b29-9739-68f74dcd611d","Type":"ContainerStarted","Data":"afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af"}
Sep 15 09:43:12.344979 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:12.344932    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc" podStartSLOduration=2.344918752 podStartE2EDuration="2.344918752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:43:12.344187521 +0000 UTC m=+1756.173979426" watchObservedRunningTime="2026-09-15 09:43:12.344918752 +0000 UTC m=+1756.174710656"
Sep 15 09:43:16.005079 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.005031    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5" podUID="bc423024-7539-4d80-8ecc-4a41be184b31" containerName="main" containerID="cri-o://6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da" gracePeriod=45
Sep 15 09:43:16.250818 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.250786    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:43:16.348806 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.348719    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc423024-7539-4d80-8ecc-4a41be184b31" containerID="6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da" exitCode=0
Sep 15 09:43:16.348806 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.348758    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"
Sep 15 09:43:16.348992 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.348807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5" event={"ID":"bc423024-7539-4d80-8ecc-4a41be184b31","Type":"ContainerDied","Data":"6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da"}
Sep 15 09:43:16.348992 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.348847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5" event={"ID":"bc423024-7539-4d80-8ecc-4a41be184b31","Type":"ContainerDied","Data":"b03c835f8aa5073a09e5cd45cbb1e582a8751b848b94387f6e28f9c3bbedc3c5"}
Sep 15 09:43:16.348992 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.348866    2579 scope.go:117] "RemoveContainer" containerID="6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da"
Sep 15 09:43:16.357167 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.357148    2579 scope.go:117] "RemoveContainer" containerID="6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da"
Sep 15 09:43:16.357447 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:43:16.357429    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da\": container with ID starting with 6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da not found: ID does not exist" containerID="6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da"
Sep 15 09:43:16.357496 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.357459    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da"} err="failed to get container status \"6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da\": rpc error: code = NotFound desc = could not find container \"6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da\": container with ID starting with 6b5afac899de391388ab0acfcb81753bc5dec2f9f6ddc675c143228c620056da not found: ID does not exist"
Sep 15 09:43:16.410773 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.410754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-dshm\") pod \"bc423024-7539-4d80-8ecc-4a41be184b31\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") "
Sep 15 09:43:16.410850 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.410792    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-model-cache\") pod \"bc423024-7539-4d80-8ecc-4a41be184b31\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") "
Sep 15 09:43:16.410892 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.410848    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-home\") pod \"bc423024-7539-4d80-8ecc-4a41be184b31\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") "
Sep 15 09:43:16.410942 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.410929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-tmp-dir\") pod \"bc423024-7539-4d80-8ecc-4a41be184b31\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") "
Sep 15 09:43:16.410982 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.410970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kr5t\" (UniqueName: \"kubernetes.io/projected/bc423024-7539-4d80-8ecc-4a41be184b31-kube-api-access-4kr5t\") pod \"bc423024-7539-4d80-8ecc-4a41be184b31\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") "
Sep 15 09:43:16.411022 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.411008    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc423024-7539-4d80-8ecc-4a41be184b31-tls-certs\") pod \"bc423024-7539-4d80-8ecc-4a41be184b31\" (UID: \"bc423024-7539-4d80-8ecc-4a41be184b31\") "
Sep 15 09:43:16.411096 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.411071    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc423024-7539-4d80-8ecc-4a41be184b31" (UID: "bc423024-7539-4d80-8ecc-4a41be184b31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:43:16.411203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.411176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc423024-7539-4d80-8ecc-4a41be184b31" (UID: "bc423024-7539-4d80-8ecc-4a41be184b31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:43:16.411203 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.411182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-home" (OuterVolumeSpecName: "home") pod "bc423024-7539-4d80-8ecc-4a41be184b31" (UID: "bc423024-7539-4d80-8ecc-4a41be184b31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:43:16.411395 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.411299    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:43:16.411395 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.411311    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:43:16.411395 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.411323    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:43:16.412973 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.412953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-dshm" (OuterVolumeSpecName: "dshm") pod "bc423024-7539-4d80-8ecc-4a41be184b31" (UID: "bc423024-7539-4d80-8ecc-4a41be184b31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:43:16.413047 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.413024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc423024-7539-4d80-8ecc-4a41be184b31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc423024-7539-4d80-8ecc-4a41be184b31" (UID: "bc423024-7539-4d80-8ecc-4a41be184b31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:43:16.413149 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.413132    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc423024-7539-4d80-8ecc-4a41be184b31-kube-api-access-4kr5t" (OuterVolumeSpecName: "kube-api-access-4kr5t") pod "bc423024-7539-4d80-8ecc-4a41be184b31" (UID: "bc423024-7539-4d80-8ecc-4a41be184b31"). InnerVolumeSpecName "kube-api-access-4kr5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:43:16.511922 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.511892    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kr5t\" (UniqueName: \"kubernetes.io/projected/bc423024-7539-4d80-8ecc-4a41be184b31-kube-api-access-4kr5t\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:43:16.511922 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.511918    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc423024-7539-4d80-8ecc-4a41be184b31-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:43:16.511922 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.511928    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc423024-7539-4d80-8ecc-4a41be184b31-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:43:16.674884 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.674857    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"]
Sep 15 09:43:16.674884 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:16.674890    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/auth-test-model-864b9a-kserve-686f8c7b67-8qpt5"]
Sep 15 09:43:18.675111 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:18.675071    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc423024-7539-4d80-8ecc-4a41be184b31" path="/var/lib/kubelet/pods/bc423024-7539-4d80-8ecc-4a41be184b31/volumes"
Sep 15 09:43:20.306397 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:20.306358    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:43:21.018752 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:21.018721    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:21.018959 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:21.018768    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:21.031326 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:21.031299    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:21.388298 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:21.388217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:43:37.499279 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.499245    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"]
Sep 15 09:43:37.499832 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.499811    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc423024-7539-4d80-8ecc-4a41be184b31" containerName="main"
Sep 15 09:43:37.499928 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.499835    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc423024-7539-4d80-8ecc-4a41be184b31" containerName="main"
Sep 15 09:43:37.499988 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.499941    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc423024-7539-4d80-8ecc-4a41be184b31" containerName="main"
Sep 15 09:43:37.502946 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.502926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.505983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.505964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b283ee\"/\"openshift-service-ca.crt\""
Sep 15 09:43:37.505983 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.505970    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b283ee\"/\"default-dockercfg-nm2c2\""
Sep 15 09:43:37.506163 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.506022    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b283ee\"/\"kube-root-ca.crt\""
Sep 15 09:43:37.506163 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.506131    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b283ee\"/\"sub-test-model-b283ee-kserve-self-signed-certs\""
Sep 15 09:43:37.512950 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.512932    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"]
Sep 15 09:43:37.588414 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.588386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-dshm\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.588556 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.588423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-model-cache\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.588556 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.588455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktbl\" (UniqueName: \"kubernetes.io/projected/f200c5a0-369b-4168-bf57-1c6529d29666-kube-api-access-7ktbl\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.588556 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.588507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f200c5a0-369b-4168-bf57-1c6529d29666-tls-certs\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.588697 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.588592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-home\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.588697 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.588636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-tmp-dir\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.689275 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.689250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktbl\" (UniqueName: \"kubernetes.io/projected/f200c5a0-369b-4168-bf57-1c6529d29666-kube-api-access-7ktbl\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.689457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.689285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f200c5a0-369b-4168-bf57-1c6529d29666-tls-certs\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.689457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.689319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-home\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.689457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.689380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-tmp-dir\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.689457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.689443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-dshm\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.689678 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.689460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-model-cache\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.689821 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.689795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-home\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.689888 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.689843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-tmp-dir\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.689955 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.689938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-model-cache\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.691738 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.691709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-dshm\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.691947 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.691931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f200c5a0-369b-4168-bf57-1c6529d29666-tls-certs\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.696038 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.696015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktbl\" (UniqueName: \"kubernetes.io/projected/f200c5a0-369b-4168-bf57-1c6529d29666-kube-api-access-7ktbl\") pod \"sub-test-model-b283ee-kserve-68999f8b8f-pfq4k\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:37.813179 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:37.813121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:38.081438 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:38.081413    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"]
Sep 15 09:43:38.083907 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:43:38.083880    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf200c5a0_369b_4168_bf57_1c6529d29666.slice/crio-b72c1b353fbeb55b2dee4a39cf1d359a8b47f80752dadbb87880afcafb883c45 WatchSource:0}: Error finding container b72c1b353fbeb55b2dee4a39cf1d359a8b47f80752dadbb87880afcafb883c45: Status 404 returned error can't find the container with id b72c1b353fbeb55b2dee4a39cf1d359a8b47f80752dadbb87880afcafb883c45
Sep 15 09:43:38.457405 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:38.457372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k" event={"ID":"f200c5a0-369b-4168-bf57-1c6529d29666","Type":"ContainerStarted","Data":"743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65"}
Sep 15 09:43:38.457551 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:38.457422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k" event={"ID":"f200c5a0-369b-4168-bf57-1c6529d29666","Type":"ContainerStarted","Data":"b72c1b353fbeb55b2dee4a39cf1d359a8b47f80752dadbb87880afcafb883c45"}
Sep 15 09:43:38.474390 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:38.474324    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k" podStartSLOduration=1.474309153 podStartE2EDuration="1.474309153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:43:38.472656849 +0000 UTC m=+1782.302448751" watchObservedRunningTime="2026-09-15 09:43:38.474309153 +0000 UTC m=+1782.304101055"
Sep 15 09:43:40.999930 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:40.999895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:43:47.813529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:47.813493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:47.813937 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:47.813666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:47.825973 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:47.825950    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:48.515250 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:48.515216    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:43:56.098307 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:43:56.098262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:44:06.566453 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:06.566417    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7558-93677b"]
Sep 15 09:44:06.569905 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:06.569888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-93677b"
Sep 15 09:44:06.572536 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:06.572510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-88s6f\""
Sep 15 09:44:06.583548 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:06.583520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7558-93677b"]
Sep 15 09:44:06.747424 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:06.747391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhkg\" (UniqueName: \"kubernetes.io/projected/7fc48a03-8b68-45b1-8bb9-d08f96b31b77-kube-api-access-xqhkg\") pod \"test-curl-7558-93677b\" (UID: \"7fc48a03-8b68-45b1-8bb9-d08f96b31b77\") " pod="opendatahub/test-curl-7558-93677b"
Sep 15 09:44:06.848838 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:06.848747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhkg\" (UniqueName: \"kubernetes.io/projected/7fc48a03-8b68-45b1-8bb9-d08f96b31b77-kube-api-access-xqhkg\") pod \"test-curl-7558-93677b\" (UID: \"7fc48a03-8b68-45b1-8bb9-d08f96b31b77\") " pod="opendatahub/test-curl-7558-93677b"
Sep 15 09:44:06.856528 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:06.856496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhkg\" (UniqueName: \"kubernetes.io/projected/7fc48a03-8b68-45b1-8bb9-d08f96b31b77-kube-api-access-xqhkg\") pod \"test-curl-7558-93677b\" (UID: \"7fc48a03-8b68-45b1-8bb9-d08f96b31b77\") " pod="opendatahub/test-curl-7558-93677b"
Sep 15 09:44:06.883470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:06.883445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-93677b"
Sep 15 09:44:07.151878 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:07.151848    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7558-93677b"]
Sep 15 09:44:07.154037 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:44:07.154010    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fc48a03_8b68_45b1_8bb9_d08f96b31b77.slice/crio-995ac95da7fa1034cdb8778ee7a0f9db6e8c03e79b3233436d14e9487dca0cd4 WatchSource:0}: Error finding container 995ac95da7fa1034cdb8778ee7a0f9db6e8c03e79b3233436d14e9487dca0cd4: Status 404 returned error can't find the container with id 995ac95da7fa1034cdb8778ee7a0f9db6e8c03e79b3233436d14e9487dca0cd4
Sep 15 09:44:07.595600 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:07.595566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7558-93677b" event={"ID":"7fc48a03-8b68-45b1-8bb9-d08f96b31b77","Type":"ContainerStarted","Data":"df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2"}
Sep 15 09:44:07.595962 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:07.595610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7558-93677b" event={"ID":"7fc48a03-8b68-45b1-8bb9-d08f96b31b77","Type":"ContainerStarted","Data":"995ac95da7fa1034cdb8778ee7a0f9db6e8c03e79b3233436d14e9487dca0cd4"}
Sep 15 09:44:07.609320 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:07.609272    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7558-93677b" podStartSLOduration=1.6092567789999999 podStartE2EDuration="1.609256779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:44:07.608435122 +0000 UTC m=+1811.438227026" watchObservedRunningTime="2026-09-15 09:44:07.609256779 +0000 UTC m=+1811.439048682"
Sep 15 09:44:08.164037 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.163998    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7558-93677b"]
Sep 15 09:44:08.167980 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.167950    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7558-93677b"]
Sep 15 09:44:08.281018 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.280952    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7558-dee343"]
Sep 15 09:44:08.281368 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.281354    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc48a03-8b68-45b1-8bb9-d08f96b31b77" containerName="test-curl-7558-93677b"
Sep 15 09:44:08.281425 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.281369    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc48a03-8b68-45b1-8bb9-d08f96b31b77" containerName="test-curl-7558-93677b"
Sep 15 09:44:08.281459 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.281441    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc48a03-8b68-45b1-8bb9-d08f96b31b77" containerName="test-curl-7558-93677b"
Sep 15 09:44:08.284502 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.284486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-dee343"
Sep 15 09:44:08.289672 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.289652    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7558-dee343"]
Sep 15 09:44:08.463290 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.463256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fksqn\" (UniqueName: \"kubernetes.io/projected/e137f77e-fd53-421e-9ac0-7b4d94c02904-kube-api-access-fksqn\") pod \"test-curl-7558-dee343\" (UID: \"e137f77e-fd53-421e-9ac0-7b4d94c02904\") " pod="opendatahub/test-curl-7558-dee343"
Sep 15 09:44:08.563772 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.563702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fksqn\" (UniqueName: \"kubernetes.io/projected/e137f77e-fd53-421e-9ac0-7b4d94c02904-kube-api-access-fksqn\") pod \"test-curl-7558-dee343\" (UID: \"e137f77e-fd53-421e-9ac0-7b4d94c02904\") " pod="opendatahub/test-curl-7558-dee343"
Sep 15 09:44:08.571817 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.571794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fksqn\" (UniqueName: \"kubernetes.io/projected/e137f77e-fd53-421e-9ac0-7b4d94c02904-kube-api-access-fksqn\") pod \"test-curl-7558-dee343\" (UID: \"e137f77e-fd53-421e-9ac0-7b4d94c02904\") " pod="opendatahub/test-curl-7558-dee343"
Sep 15 09:44:08.595546 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.595523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-dee343"
Sep 15 09:44:08.601073 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:44:08.601055    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7558-93677b" container="test-curl-7558-93677b"
Sep 15 09:44:08.861519 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:08.861498    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7558-dee343"]
Sep 15 09:44:08.863662 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:44:08.863635    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode137f77e_fd53_421e_9ac0_7b4d94c02904.slice/crio-1bf79e6c617d2a6000925911ab3738d37ec7470cc112e1e6362e42daba364a7f WatchSource:0}: Error finding container 1bf79e6c617d2a6000925911ab3738d37ec7470cc112e1e6362e42daba364a7f: Status 404 returned error can't find the container with id 1bf79e6c617d2a6000925911ab3738d37ec7470cc112e1e6362e42daba364a7f
Sep 15 09:44:09.607391 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:09.607354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7558-dee343" event={"ID":"e137f77e-fd53-421e-9ac0-7b4d94c02904","Type":"ContainerStarted","Data":"d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181"}
Sep 15 09:44:09.607811 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:09.607401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7558-dee343" event={"ID":"e137f77e-fd53-421e-9ac0-7b4d94c02904","Type":"ContainerStarted","Data":"1bf79e6c617d2a6000925911ab3738d37ec7470cc112e1e6362e42daba364a7f"}
Sep 15 09:44:09.607811 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:09.607473    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7558-93677b" podUID="7fc48a03-8b68-45b1-8bb9-d08f96b31b77" containerName="test-curl-7558-93677b" containerID="cri-o://df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2" gracePeriod=2
Sep 15 09:44:09.619749 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:09.619702    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7558-dee343" podStartSLOduration=1.6196809509999999 podStartE2EDuration="1.619680951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:44:09.619003786 +0000 UTC m=+1813.448795705" watchObservedRunningTime="2026-09-15 09:44:09.619680951 +0000 UTC m=+1813.449472855"
Sep 15 09:44:10.173318 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:10.173279    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7558-dee343"]
Sep 15 09:44:10.180005 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:10.178449    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7558-dee343"]
Sep 15 09:44:10.612636 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:44:10.612607    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7558-dee343" container="test-curl-7558-dee343"
Sep 15 09:44:11.391792 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:11.391759    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:44:11.617237 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:11.617203    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7558-dee343" podUID="e137f77e-fd53-421e-9ac0-7b4d94c02904" containerName="test-curl-7558-dee343" containerID="cri-o://d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181" gracePeriod=2
Sep 15 09:44:11.790364 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:11.790327    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-93677b"
Sep 15 09:44:11.890273 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:11.890248    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqhkg\" (UniqueName: \"kubernetes.io/projected/7fc48a03-8b68-45b1-8bb9-d08f96b31b77-kube-api-access-xqhkg\") pod \"7fc48a03-8b68-45b1-8bb9-d08f96b31b77\" (UID: \"7fc48a03-8b68-45b1-8bb9-d08f96b31b77\") "
Sep 15 09:44:11.892240 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:11.892218    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc48a03-8b68-45b1-8bb9-d08f96b31b77-kube-api-access-xqhkg" (OuterVolumeSpecName: "kube-api-access-xqhkg") pod "7fc48a03-8b68-45b1-8bb9-d08f96b31b77" (UID: "7fc48a03-8b68-45b1-8bb9-d08f96b31b77"). InnerVolumeSpecName "kube-api-access-xqhkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:44:11.991269 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:11.991244    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqhkg\" (UniqueName: \"kubernetes.io/projected/7fc48a03-8b68-45b1-8bb9-d08f96b31b77-kube-api-access-xqhkg\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:12.283715 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:12.283644    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"]
Sep 15 09:44:12.623285 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:12.623205    2579 generic.go:356] "Generic (PLEG): container finished" podID="7fc48a03-8b68-45b1-8bb9-d08f96b31b77" containerID="df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2" exitCode=137
Sep 15 09:44:12.623285 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:12.623248    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-93677b"
Sep 15 09:44:12.623285 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:12.623261    2579 scope.go:117] "RemoveContainer" containerID="df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2"
Sep 15 09:44:12.631759 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:12.631738    2579 scope.go:117] "RemoveContainer" containerID="df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2"
Sep 15 09:44:12.632123 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:44:12.632083    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2\": container with ID starting with df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2 not found: ID does not exist" containerID="df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2"
Sep 15 09:44:12.632236 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:12.632133    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2"} err="failed to get container status \"df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2\": rpc error: code = NotFound desc = could not find container \"df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2\": container with ID starting with df8af6c92b60e9bdf60f1386e25a309b2f3883da6a994cafef10cc23fca1d2a2 not found: ID does not exist"
Sep 15 09:44:12.634046 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:44:12.634021    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7558-93677b\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="7fc48a03-8b68-45b1-8bb9-d08f96b31b77" pod="opendatahub/test-curl-7558-93677b"
Sep 15 09:44:12.674433 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:12.674414    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fc48a03-8b68-45b1-8bb9-d08f96b31b77" path="/var/lib/kubelet/pods/7fc48a03-8b68-45b1-8bb9-d08f96b31b77/volumes"
Sep 15 09:44:13.284470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:13.284438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"]
Sep 15 09:44:13.781230 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:13.781209    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-dee343"
Sep 15 09:44:13.809479 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:13.809456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fksqn\" (UniqueName: \"kubernetes.io/projected/e137f77e-fd53-421e-9ac0-7b4d94c02904-kube-api-access-fksqn\") pod \"e137f77e-fd53-421e-9ac0-7b4d94c02904\" (UID: \"e137f77e-fd53-421e-9ac0-7b4d94c02904\") "
Sep 15 09:44:13.811313 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:13.811294    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e137f77e-fd53-421e-9ac0-7b4d94c02904-kube-api-access-fksqn" (OuterVolumeSpecName: "kube-api-access-fksqn") pod "e137f77e-fd53-421e-9ac0-7b4d94c02904" (UID: "e137f77e-fd53-421e-9ac0-7b4d94c02904"). InnerVolumeSpecName "kube-api-access-fksqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:44:13.910913 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:13.910885    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fksqn\" (UniqueName: \"kubernetes.io/projected/e137f77e-fd53-421e-9ac0-7b4d94c02904-kube-api-access-fksqn\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:14.637592 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:14.637555    2579 generic.go:356] "Generic (PLEG): container finished" podID="e137f77e-fd53-421e-9ac0-7b4d94c02904" containerID="d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181" exitCode=137
Sep 15 09:44:14.637771 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:14.637597    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7558-dee343"
Sep 15 09:44:14.637771 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:14.637634    2579 scope.go:117] "RemoveContainer" containerID="d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181"
Sep 15 09:44:14.646702 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:14.646684    2579 scope.go:117] "RemoveContainer" containerID="d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181"
Sep 15 09:44:14.646960 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:44:14.646941    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181\": container with ID starting with d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181 not found: ID does not exist" containerID="d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181"
Sep 15 09:44:14.647005 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:14.646967    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181"} err="failed to get container status \"d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181\": rpc error: code = NotFound desc = could not find container \"d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181\": container with ID starting with d920629b31ef92215f86236d204740c1a1d488cadedc242b63512ae3fdc81181 not found: ID does not exist"
Sep 15 09:44:14.647610 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:44:14.647592    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7558-dee343\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="e137f77e-fd53-421e-9ac0-7b4d94c02904" pod="opendatahub/test-curl-7558-dee343"
Sep 15 09:44:14.675482 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:14.675458    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e137f77e-fd53-421e-9ac0-7b4d94c02904" path="/var/lib/kubelet/pods/e137f77e-fd53-421e-9ac0-7b4d94c02904/volumes"
Sep 15 09:44:21.500727 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.500688    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"]
Sep 15 09:44:21.501109 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.501095    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e137f77e-fd53-421e-9ac0-7b4d94c02904" containerName="test-curl-7558-dee343"
Sep 15 09:44:21.501154 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.501111    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e137f77e-fd53-421e-9ac0-7b4d94c02904" containerName="test-curl-7558-dee343"
Sep 15 09:44:21.501194 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.501187    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e137f77e-fd53-421e-9ac0-7b4d94c02904" containerName="test-curl-7558-dee343"
Sep 15 09:44:21.505886 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.505865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.507951 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.507929    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-864b9a\"/\"rate-test-model-864b9a-kserve-self-signed-certs\""
Sep 15 09:44:21.514176 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.514155    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"]
Sep 15 09:44:21.578426 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.578400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-dshm\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.578543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.578433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0866312-2bf2-4381-b468-af4cb87c84d3-tls-certs\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.578543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.578460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfwz4\" (UniqueName: \"kubernetes.io/projected/b0866312-2bf2-4381-b468-af4cb87c84d3-kube-api-access-gfwz4\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.578543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.578531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-home\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.578648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.578575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-tmp-dir\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.578648 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.578603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-model-cache\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.679170 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.679148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-dshm\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.679291 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.679178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0866312-2bf2-4381-b468-af4cb87c84d3-tls-certs\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.679291 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.679198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfwz4\" (UniqueName: \"kubernetes.io/projected/b0866312-2bf2-4381-b468-af4cb87c84d3-kube-api-access-gfwz4\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.679291 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.679229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-home\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.679291 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.679264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-tmp-dir\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.679507 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.679301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-model-cache\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.679642 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.679621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-home\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.679747 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.679725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-tmp-dir\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.679799 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.679762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-model-cache\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.681506 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.681480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-dshm\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.681756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.681737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0866312-2bf2-4381-b468-af4cb87c84d3-tls-certs\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.686482 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.686461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfwz4\" (UniqueName: \"kubernetes.io/projected/b0866312-2bf2-4381-b468-af4cb87c84d3-kube-api-access-gfwz4\") pod \"rate-test-model-864b9a-kserve-7c5b597fbc-pzndf\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") " pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.816611 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.816544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:21.988647 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:21.988620    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:44:22.086031 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:22.086008    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"]
Sep 15 09:44:22.088136 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:44:22.088108    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0866312_2bf2_4381_b468_af4cb87c84d3.slice/crio-3930257bcbac3004ebc69f29843c8f7c17fe637b8d5aa319b2f28205e9134e1e WatchSource:0}: Error finding container 3930257bcbac3004ebc69f29843c8f7c17fe637b8d5aa319b2f28205e9134e1e: Status 404 returned error can't find the container with id 3930257bcbac3004ebc69f29843c8f7c17fe637b8d5aa319b2f28205e9134e1e
Sep 15 09:44:22.684038 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:22.684005    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf" event={"ID":"b0866312-2bf2-4381-b468-af4cb87c84d3","Type":"ContainerStarted","Data":"9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513"}
Sep 15 09:44:22.684419 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:22.684045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf" event={"ID":"b0866312-2bf2-4381-b468-af4cb87c84d3","Type":"ContainerStarted","Data":"3930257bcbac3004ebc69f29843c8f7c17fe637b8d5aa319b2f28205e9134e1e"}
Sep 15 09:44:22.700222 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:22.700171    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf" podStartSLOduration=1.700153286 podStartE2EDuration="1.700153286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:44:22.699526519 +0000 UTC m=+1826.529318424" watchObservedRunningTime="2026-09-15 09:44:22.700153286 +0000 UTC m=+1826.529945189"
Sep 15 09:44:27.303120 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.303048    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc" podUID="1bf7611a-de18-4b29-9739-68f74dcd611d" containerName="main" containerID="cri-o://afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af" gracePeriod=45
Sep 15 09:44:27.555359 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.555284    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:44:27.629394 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629358    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-dshm\") pod \"1bf7611a-de18-4b29-9739-68f74dcd611d\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") "
Sep 15 09:44:27.629529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629425    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf7611a-de18-4b29-9739-68f74dcd611d-tls-certs\") pod \"1bf7611a-de18-4b29-9739-68f74dcd611d\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") "
Sep 15 09:44:27.629529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629464    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-model-cache\") pod \"1bf7611a-de18-4b29-9739-68f74dcd611d\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") "
Sep 15 09:44:27.629529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629484    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smxpf\" (UniqueName: \"kubernetes.io/projected/1bf7611a-de18-4b29-9739-68f74dcd611d-kube-api-access-smxpf\") pod \"1bf7611a-de18-4b29-9739-68f74dcd611d\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") "
Sep 15 09:44:27.629673 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-tmp-dir\") pod \"1bf7611a-de18-4b29-9739-68f74dcd611d\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") "
Sep 15 09:44:27.629673 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-home\") pod \"1bf7611a-de18-4b29-9739-68f74dcd611d\" (UID: \"1bf7611a-de18-4b29-9739-68f74dcd611d\") "
Sep 15 09:44:27.629780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bf7611a-de18-4b29-9739-68f74dcd611d" (UID: "1bf7611a-de18-4b29-9739-68f74dcd611d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:44:27.629853 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bf7611a-de18-4b29-9739-68f74dcd611d" (UID: "1bf7611a-de18-4b29-9739-68f74dcd611d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:44:27.629853 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629845    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:27.629959 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.629892    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-home" (OuterVolumeSpecName: "home") pod "1bf7611a-de18-4b29-9739-68f74dcd611d" (UID: "1bf7611a-de18-4b29-9739-68f74dcd611d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:44:27.631547 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.631519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf7611a-de18-4b29-9739-68f74dcd611d-kube-api-access-smxpf" (OuterVolumeSpecName: "kube-api-access-smxpf") pod "1bf7611a-de18-4b29-9739-68f74dcd611d" (UID: "1bf7611a-de18-4b29-9739-68f74dcd611d"). InnerVolumeSpecName "kube-api-access-smxpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:44:27.631662 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.631551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf7611a-de18-4b29-9739-68f74dcd611d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bf7611a-de18-4b29-9739-68f74dcd611d" (UID: "1bf7611a-de18-4b29-9739-68f74dcd611d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:44:27.631662 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.631576    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-dshm" (OuterVolumeSpecName: "dshm") pod "1bf7611a-de18-4b29-9739-68f74dcd611d" (UID: "1bf7611a-de18-4b29-9739-68f74dcd611d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:44:27.708756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.708723    2579 generic.go:356] "Generic (PLEG): container finished" podID="1bf7611a-de18-4b29-9739-68f74dcd611d" containerID="afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af" exitCode=0
Sep 15 09:44:27.708879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.708765    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"
Sep 15 09:44:27.708879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.708811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc" event={"ID":"1bf7611a-de18-4b29-9739-68f74dcd611d","Type":"ContainerDied","Data":"afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af"}
Sep 15 09:44:27.708879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.708856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc" event={"ID":"1bf7611a-de18-4b29-9739-68f74dcd611d","Type":"ContainerDied","Data":"26036769dd9f9c82ba00927dc3bc41345c4fa2d11c63c51e173f9b621b11b695"}
Sep 15 09:44:27.708879 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.708875    2579 scope.go:117] "RemoveContainer" containerID="afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af"
Sep 15 09:44:27.717382 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.717361    2579 scope.go:117] "RemoveContainer" containerID="afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af"
Sep 15 09:44:27.717665 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:44:27.717646    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af\": container with ID starting with afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af not found: ID does not exist" containerID="afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af"
Sep 15 09:44:27.717722 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.717674    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af"} err="failed to get container status \"afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af\": rpc error: code = NotFound desc = could not find container \"afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af\": container with ID starting with afc397cbd17a3b18734e1058b05711228546c2d3b54f7a4e651145f41f2889af not found: ID does not exist"
Sep 15 09:44:27.728748 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.728729    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"]
Sep 15 09:44:27.731066 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.731048    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:27.731066 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.731066    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:27.731190 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.731076    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf7611a-de18-4b29-9739-68f74dcd611d-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:27.731190 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.731084    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf7611a-de18-4b29-9739-68f74dcd611d-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:27.731190 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.731092    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxpf\" (UniqueName: \"kubernetes.io/projected/1bf7611a-de18-4b29-9739-68f74dcd611d-kube-api-access-smxpf\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:27.731611 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:27.731594    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/sub-test-model-864b9a-kserve-68867c44c-c4mtc"]
Sep 15 09:44:28.303769 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.303730    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k" podUID="f200c5a0-369b-4168-bf57-1c6529d29666" containerName="main" containerID="cri-o://743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65" gracePeriod=45
Sep 15 09:44:28.545979 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.545957    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:44:28.637457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f200c5a0-369b-4168-bf57-1c6529d29666-tls-certs\") pod \"f200c5a0-369b-4168-bf57-1c6529d29666\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") "
Sep 15 09:44:28.637457 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-model-cache\") pod \"f200c5a0-369b-4168-bf57-1c6529d29666\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") "
Sep 15 09:44:28.637610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637476    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-tmp-dir\") pod \"f200c5a0-369b-4168-bf57-1c6529d29666\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") "
Sep 15 09:44:28.637610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637558    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-home\") pod \"f200c5a0-369b-4168-bf57-1c6529d29666\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") "
Sep 15 09:44:28.637610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ktbl\" (UniqueName: \"kubernetes.io/projected/f200c5a0-369b-4168-bf57-1c6529d29666-kube-api-access-7ktbl\") pod \"f200c5a0-369b-4168-bf57-1c6529d29666\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") "
Sep 15 09:44:28.637758 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637616    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-dshm\") pod \"f200c5a0-369b-4168-bf57-1c6529d29666\" (UID: \"f200c5a0-369b-4168-bf57-1c6529d29666\") "
Sep 15 09:44:28.637758 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637743    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-model-cache" (OuterVolumeSpecName: "model-cache") pod "f200c5a0-369b-4168-bf57-1c6529d29666" (UID: "f200c5a0-369b-4168-bf57-1c6529d29666"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:44:28.637875 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637778    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f200c5a0-369b-4168-bf57-1c6529d29666" (UID: "f200c5a0-369b-4168-bf57-1c6529d29666"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:44:28.637875 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637839    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-home" (OuterVolumeSpecName: "home") pod "f200c5a0-369b-4168-bf57-1c6529d29666" (UID: "f200c5a0-369b-4168-bf57-1c6529d29666"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:44:28.637996 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.637974    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:28.638039 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.638000    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:28.638039 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.638012    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:28.639679 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.639655    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-dshm" (OuterVolumeSpecName: "dshm") pod "f200c5a0-369b-4168-bf57-1c6529d29666" (UID: "f200c5a0-369b-4168-bf57-1c6529d29666"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:44:28.639785 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.639720    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f200c5a0-369b-4168-bf57-1c6529d29666-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f200c5a0-369b-4168-bf57-1c6529d29666" (UID: "f200c5a0-369b-4168-bf57-1c6529d29666"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:44:28.639827 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.639815    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f200c5a0-369b-4168-bf57-1c6529d29666-kube-api-access-7ktbl" (OuterVolumeSpecName: "kube-api-access-7ktbl") pod "f200c5a0-369b-4168-bf57-1c6529d29666" (UID: "f200c5a0-369b-4168-bf57-1c6529d29666"). InnerVolumeSpecName "kube-api-access-7ktbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:44:28.675289 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.675263    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf7611a-de18-4b29-9739-68f74dcd611d" path="/var/lib/kubelet/pods/1bf7611a-de18-4b29-9739-68f74dcd611d/volumes"
Sep 15 09:44:28.714766 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.714738    2579 generic.go:356] "Generic (PLEG): container finished" podID="f200c5a0-369b-4168-bf57-1c6529d29666" containerID="743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65" exitCode=0
Sep 15 09:44:28.714873 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.714772    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"
Sep 15 09:44:28.714873 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.714825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k" event={"ID":"f200c5a0-369b-4168-bf57-1c6529d29666","Type":"ContainerDied","Data":"743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65"}
Sep 15 09:44:28.714873 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.714855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k" event={"ID":"f200c5a0-369b-4168-bf57-1c6529d29666","Type":"ContainerDied","Data":"b72c1b353fbeb55b2dee4a39cf1d359a8b47f80752dadbb87880afcafb883c45"}
Sep 15 09:44:28.714873 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.714871    2579 scope.go:117] "RemoveContainer" containerID="743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65"
Sep 15 09:44:28.722831 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.722812    2579 scope.go:117] "RemoveContainer" containerID="743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65"
Sep 15 09:44:28.723087 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:44:28.723069    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65\": container with ID starting with 743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65 not found: ID does not exist" containerID="743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65"
Sep 15 09:44:28.723134 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.723093    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65"} err="failed to get container status \"743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65\": rpc error: code = NotFound desc = could not find container \"743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65\": container with ID starting with 743f85122acf9f1b0a6166bccccfcc2517affc2ed6d07e190e8ade0c69832f65 not found: ID does not exist"
Sep 15 09:44:28.729249 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.729229    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"]
Sep 15 09:44:28.734521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.734501    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k"]
Sep 15 09:44:28.738510 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.738484    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktbl\" (UniqueName: \"kubernetes.io/projected/f200c5a0-369b-4168-bf57-1c6529d29666-kube-api-access-7ktbl\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:28.738510 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.738506    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f200c5a0-369b-4168-bf57-1c6529d29666-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:28.738657 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:28.738517    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f200c5a0-369b-4168-bf57-1c6529d29666-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:44:30.675647 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:30.675614    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f200c5a0-369b-4168-bf57-1c6529d29666" path="/var/lib/kubelet/pods/f200c5a0-369b-4168-bf57-1c6529d29666/volumes"
Sep 15 09:44:31.090733 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:31.090651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:44:31.817152 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:31.817118    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:31.817569 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:31.817165    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:31.829659 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:31.829636    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:32.743616 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:32.743587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:44:33.505043 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:33.504999    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-b283ee/sub-test-model-b283ee-kserve-68999f8b8f-pfq4k" podUID="f200c5a0-369b-4168-bf57-1c6529d29666" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Sep 15 09:44:49.093924 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.093893    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"]
Sep 15 09:44:49.094380 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.094251    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf7611a-de18-4b29-9739-68f74dcd611d" containerName="main"
Sep 15 09:44:49.094380 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.094262    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf7611a-de18-4b29-9739-68f74dcd611d" containerName="main"
Sep 15 09:44:49.094380 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.094278    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f200c5a0-369b-4168-bf57-1c6529d29666" containerName="main"
Sep 15 09:44:49.094380 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.094283    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f200c5a0-369b-4168-bf57-1c6529d29666" containerName="main"
Sep 15 09:44:49.094380 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.094368    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf7611a-de18-4b29-9739-68f74dcd611d" containerName="main"
Sep 15 09:44:49.094380 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.094381    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f200c5a0-369b-4168-bf57-1c6529d29666" containerName="main"
Sep 15 09:44:49.097618 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.097599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.099780 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.099752    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b283ee\"/\"default-dockercfg-nm2c2\""
Sep 15 09:44:49.100589 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.100559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b283ee\"/\"rate-test-model-b283ee-kserve-self-signed-certs\""
Sep 15 09:44:49.100711 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.100631    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b283ee\"/\"openshift-service-ca.crt\""
Sep 15 09:44:49.100711 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.100635    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b283ee\"/\"kube-root-ca.crt\""
Sep 15 09:44:49.107030 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.107006    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"]
Sep 15 09:44:49.219189 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.219160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tmp-dir\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.219321 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.219197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tls-certs\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.219321 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.219291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-model-cache\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.219436 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.219330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-dshm\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.219436 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.219372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nsc\" (UniqueName: \"kubernetes.io/projected/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-kube-api-access-98nsc\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.219436 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.219426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-home\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.319808 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.319781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tmp-dir\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.319987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.319818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tls-certs\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.319987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.319864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-model-cache\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.319987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.319911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-dshm\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.319987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.319954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nsc\" (UniqueName: \"kubernetes.io/projected/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-kube-api-access-98nsc\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.319987 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.319983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-home\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.320274 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.320224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tmp-dir\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.320274 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.320233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-model-cache\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.320426 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.320375    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-home\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.322136 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.322111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-dshm\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.322407 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.322389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tls-certs\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.327663 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.327640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nsc\" (UniqueName: \"kubernetes.io/projected/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-kube-api-access-98nsc\") pod \"rate-test-model-b283ee-kserve-6c7944d6cf-wtff8\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") " pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.407807 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.407785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:49.679178 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.679152    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"]
Sep 15 09:44:49.681363 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:44:49.681315    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdb4fa3a_3df9_47ea_ad71_ffebbcf6ea8e.slice/crio-ed0edab101d5ada3b88c978d572d2f15673705deb781a194dc7c73dee76f2d99 WatchSource:0}: Error finding container ed0edab101d5ada3b88c978d572d2f15673705deb781a194dc7c73dee76f2d99: Status 404 returned error can't find the container with id ed0edab101d5ada3b88c978d572d2f15673705deb781a194dc7c73dee76f2d99
Sep 15 09:44:49.815578 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.815545    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8" event={"ID":"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e","Type":"ContainerStarted","Data":"16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac"}
Sep 15 09:44:49.815739 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.815587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8" event={"ID":"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e","Type":"ContainerStarted","Data":"ed0edab101d5ada3b88c978d572d2f15673705deb781a194dc7c73dee76f2d99"}
Sep 15 09:44:49.830213 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:49.830162    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8" podStartSLOduration=0.830144915 podStartE2EDuration="830.144915ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:44:49.829884178 +0000 UTC m=+1853.659676081" watchObservedRunningTime="2026-09-15 09:44:49.830144915 +0000 UTC m=+1853.659936821"
Sep 15 09:44:52.712956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:52.712921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:44:59.407917 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:59.407872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:59.408291 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:59.407939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:59.419997 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:59.419969    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:44:59.875816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:44:59.875741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:45:05.893644 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:05.893610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:45:16.291470 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:16.291436    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:45:21.609543 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:21.609454    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"]
Sep 15 09:45:22.776763 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:22.776731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"]
Sep 15 09:45:28.654152 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.654120    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"]
Sep 15 09:45:28.657668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.657646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.660003 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.659984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6593fe-serving-cert\""
Sep 15 09:45:28.679608 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.679574    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"]
Sep 15 09:45:28.766419 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.766387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.766419 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.766418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-metrics-certs\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.766604 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.766560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqr9\" (UniqueName: \"kubernetes.io/projected/c5303e6f-b116-4845-8411-3cdf0f7a89f9-kube-api-access-6jqr9\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.766604 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.766596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5303e6f-b116-4845-8411-3cdf0f7a89f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.868034 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.868002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqr9\" (UniqueName: \"kubernetes.io/projected/c5303e6f-b116-4845-8411-3cdf0f7a89f9-kube-api-access-6jqr9\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.868175 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.868038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5303e6f-b116-4845-8411-3cdf0f7a89f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.868175 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.868117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.868175 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.868147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-metrics-certs\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.868656 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.868639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5303e6f-b116-4845-8411-3cdf0f7a89f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.870479 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.870457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.870611 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.870488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-metrics-certs\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.875138 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.875118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqr9\" (UniqueName: \"kubernetes.io/projected/c5303e6f-b116-4845-8411-3cdf0f7a89f9-kube-api-access-6jqr9\") pod \"maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:28.968210 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:28.968188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:29.242178 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:29.240102    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"]
Sep 15 09:45:29.249164 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:29.249139    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:45:30.009803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:30.009719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v" event={"ID":"c5303e6f-b116-4845-8411-3cdf0f7a89f9","Type":"ContainerStarted","Data":"783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7"}
Sep 15 09:45:30.009803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:30.009767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v" event={"ID":"c5303e6f-b116-4845-8411-3cdf0f7a89f9","Type":"ContainerStarted","Data":"e71b1d46616fae6921e86d0b955edacd23e78aa808c0f6bab936c5427c1add81"}
Sep 15 09:45:30.010199 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:30.009859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:30.026012 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:30.025959    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v" podStartSLOduration=1.5885129199999999 podStartE2EDuration="2.025945553s" totalImagesPullingTime="437.432633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:45:30.025051403 +0000 UTC m=+1893.854843306" watchObservedRunningTime="2026-09-15 09:45:30.025945553 +0000 UTC m=+1893.855737459"
Sep 15 09:45:32.893302 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:32.893266    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:45:36.019982 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:36.019952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:45:36.629765 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:36.629723    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf" podUID="b0866312-2bf2-4381-b468-af4cb87c84d3" containerName="main" containerID="cri-o://9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513" gracePeriod=45
Sep 15 09:45:36.872400 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:36.872375    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:45:37.041167 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041130    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0866312-2bf2-4381-b468-af4cb87c84d3-tls-certs\") pod \"b0866312-2bf2-4381-b468-af4cb87c84d3\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") "
Sep 15 09:45:37.041601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041221    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-model-cache\") pod \"b0866312-2bf2-4381-b468-af4cb87c84d3\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") "
Sep 15 09:45:37.041601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfwz4\" (UniqueName: \"kubernetes.io/projected/b0866312-2bf2-4381-b468-af4cb87c84d3-kube-api-access-gfwz4\") pod \"b0866312-2bf2-4381-b468-af4cb87c84d3\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") "
Sep 15 09:45:37.041601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-dshm\") pod \"b0866312-2bf2-4381-b468-af4cb87c84d3\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") "
Sep 15 09:45:37.041601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041493    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-home\") pod \"b0866312-2bf2-4381-b468-af4cb87c84d3\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") "
Sep 15 09:45:37.041601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041545    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0866312-2bf2-4381-b468-af4cb87c84d3" (UID: "b0866312-2bf2-4381-b468-af4cb87c84d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:45:37.041601 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-tmp-dir\") pod \"b0866312-2bf2-4381-b468-af4cb87c84d3\" (UID: \"b0866312-2bf2-4381-b468-af4cb87c84d3\") "
Sep 15 09:45:37.041947 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0866312-2bf2-4381-b468-af4cb87c84d3" (UID: "b0866312-2bf2-4381-b468-af4cb87c84d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:45:37.041947 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-home" (OuterVolumeSpecName: "home") pod "b0866312-2bf2-4381-b468-af4cb87c84d3" (UID: "b0866312-2bf2-4381-b468-af4cb87c84d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:45:37.042048 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.041985    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:37.042048 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.042008    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:37.043569 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.043534    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0866312-2bf2-4381-b468-af4cb87c84d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0866312-2bf2-4381-b468-af4cb87c84d3" (UID: "b0866312-2bf2-4381-b468-af4cb87c84d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:45:37.043710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.043626    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0866312-2bf2-4381-b468-af4cb87c84d3-kube-api-access-gfwz4" (OuterVolumeSpecName: "kube-api-access-gfwz4") pod "b0866312-2bf2-4381-b468-af4cb87c84d3" (UID: "b0866312-2bf2-4381-b468-af4cb87c84d3"). InnerVolumeSpecName "kube-api-access-gfwz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:45:37.043710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.043628    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-dshm" (OuterVolumeSpecName: "dshm") pod "b0866312-2bf2-4381-b468-af4cb87c84d3" (UID: "b0866312-2bf2-4381-b468-af4cb87c84d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:45:37.044264 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.044241    2579 generic.go:356] "Generic (PLEG): container finished" podID="b0866312-2bf2-4381-b468-af4cb87c84d3" containerID="9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513" exitCode=0
Sep 15 09:45:37.044379 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.044291    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"
Sep 15 09:45:37.044443 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.044382    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf" event={"ID":"b0866312-2bf2-4381-b468-af4cb87c84d3","Type":"ContainerDied","Data":"9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513"}
Sep 15 09:45:37.044443 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.044431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf" event={"ID":"b0866312-2bf2-4381-b468-af4cb87c84d3","Type":"ContainerDied","Data":"3930257bcbac3004ebc69f29843c8f7c17fe637b8d5aa319b2f28205e9134e1e"}
Sep 15 09:45:37.044545 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.044450    2579 scope.go:117] "RemoveContainer" containerID="9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513"
Sep 15 09:45:37.062522 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.062503    2579 scope.go:117] "RemoveContainer" containerID="9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513"
Sep 15 09:45:37.062825 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:45:37.062808    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513\": container with ID starting with 9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513 not found: ID does not exist" containerID="9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513"
Sep 15 09:45:37.062894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.062832    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513"} err="failed to get container status \"9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513\": rpc error: code = NotFound desc = could not find container \"9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513\": container with ID starting with 9ed81f098c465baed7c20c1e718bb4b9958f573b5dfff26446ce92b1a3af0513 not found: ID does not exist"
Sep 15 09:45:37.072221 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.072196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"]
Sep 15 09:45:37.075871 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.075853    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-864b9a/rate-test-model-864b9a-kserve-7c5b597fbc-pzndf"]
Sep 15 09:45:37.142656 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.142629    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:37.142656 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.142652    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0866312-2bf2-4381-b468-af4cb87c84d3-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:37.142850 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.142664    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0866312-2bf2-4381-b468-af4cb87c84d3-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:37.142850 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.142677    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfwz4\" (UniqueName: \"kubernetes.io/projected/b0866312-2bf2-4381-b468-af4cb87c84d3-kube-api-access-gfwz4\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:37.796262 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:37.796224    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8" podUID="bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" containerName="main" containerID="cri-o://16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac" gracePeriod=45
Sep 15 09:45:38.042626 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.042596    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:45:38.050895 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.050822    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-model-cache\") pod \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") "
Sep 15 09:45:38.050895 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.050853    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98nsc\" (UniqueName: \"kubernetes.io/projected/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-kube-api-access-98nsc\") pod \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") "
Sep 15 09:45:38.050895 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.050887    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tls-certs\") pod \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") "
Sep 15 09:45:38.051146 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.050912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-home\") pod \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") "
Sep 15 09:45:38.051146 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.050993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-dshm\") pod \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") "
Sep 15 09:45:38.051146 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tmp-dir\") pod \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\" (UID: \"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e\") "
Sep 15 09:45:38.051146 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051117    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" (UID: "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:45:38.051389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051287    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:38.051389 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051321    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-home" (OuterVolumeSpecName: "home") pod "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" (UID: "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:45:38.051521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051428    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" (UID: "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:45:38.051521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051443    2579 generic.go:356] "Generic (PLEG): container finished" podID="bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" containerID="16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac" exitCode=0
Sep 15 09:45:38.051521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051492    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"
Sep 15 09:45:38.051521 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8" event={"ID":"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e","Type":"ContainerDied","Data":"16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac"}
Sep 15 09:45:38.051728 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8" event={"ID":"bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e","Type":"ContainerDied","Data":"ed0edab101d5ada3b88c978d572d2f15673705deb781a194dc7c73dee76f2d99"}
Sep 15 09:45:38.051728 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.051548    2579 scope.go:117] "RemoveContainer" containerID="16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac"
Sep 15 09:45:38.053378 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.053329    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" (UID: "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:45:38.053590 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.053569    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-dshm" (OuterVolumeSpecName: "dshm") pod "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" (UID: "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:45:38.053590 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.053581    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-kube-api-access-98nsc" (OuterVolumeSpecName: "kube-api-access-98nsc") pod "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" (UID: "bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e"). InnerVolumeSpecName "kube-api-access-98nsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:45:38.074147 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.074107    2579 scope.go:117] "RemoveContainer" containerID="16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac"
Sep 15 09:45:38.074538 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:45:38.074516    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac\": container with ID starting with 16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac not found: ID does not exist" containerID="16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac"
Sep 15 09:45:38.074631 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.074549    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac"} err="failed to get container status \"16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac\": rpc error: code = NotFound desc = could not find container \"16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac\": container with ID starting with 16b61cd6e953934c7d4c31d9fd396253e07692bdd44d8479d8f02faf5f97ddac not found: ID does not exist"
Sep 15 09:45:38.152383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.152325    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:38.152383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.152373    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:38.152383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.152382    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:38.152383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.152392    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:38.152660 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.152400    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98nsc\" (UniqueName: \"kubernetes.io/projected/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e-kube-api-access-98nsc\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:45:38.372305 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.372272    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"]
Sep 15 09:45:38.375505 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.375482    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b283ee/rate-test-model-b283ee-kserve-6c7944d6cf-wtff8"]
Sep 15 09:45:38.674842 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.674810    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0866312-2bf2-4381-b468-af4cb87c84d3" path="/var/lib/kubelet/pods/b0866312-2bf2-4381-b468-af4cb87c84d3/volumes"
Sep 15 09:45:38.675201 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:38.675188    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" path="/var/lib/kubelet/pods/bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e/volumes"
Sep 15 09:45:58.600044 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.600006    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"]
Sep 15 09:45:58.600653 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.600633    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" containerName="main"
Sep 15 09:45:58.600748 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.600655    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" containerName="main"
Sep 15 09:45:58.600748 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.600687    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0866312-2bf2-4381-b468-af4cb87c84d3" containerName="main"
Sep 15 09:45:58.600748 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.600697    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0866312-2bf2-4381-b468-af4cb87c84d3" containerName="main"
Sep 15 09:45:58.600909 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.600783    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0866312-2bf2-4381-b468-af4cb87c84d3" containerName="main"
Sep 15 09:45:58.600909 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.600798    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb4fa3a-3df9-47ea-ad71-ffebbcf6ea8e" containerName="main"
Sep 15 09:45:58.605726 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.605697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.607978 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.607943    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-527142-serving-cert\""
Sep 15 09:45:58.616721 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.616675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"]
Sep 15 09:45:58.729512 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.729475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d5e671-287c-44be-82fe-3ccfd91ead91-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.729710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.729563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-metrics-certs\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.729710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.729584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxgj\" (UniqueName: \"kubernetes.io/projected/04d5e671-287c-44be-82fe-3ccfd91ead91-kube-api-access-tbxgj\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.729710 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.729615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-maas-api-tls\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.830691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.830654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-maas-api-tls\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.830881 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.830698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d5e671-287c-44be-82fe-3ccfd91ead91-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.830881 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.830809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-metrics-certs\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.830881 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.830835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxgj\" (UniqueName: \"kubernetes.io/projected/04d5e671-287c-44be-82fe-3ccfd91ead91-kube-api-access-tbxgj\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.831532 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.831495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d5e671-287c-44be-82fe-3ccfd91ead91-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.833113 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.833081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-maas-api-tls\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.833277 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.833260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-metrics-certs\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.838356 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.838317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxgj\" (UniqueName: \"kubernetes.io/projected/04d5e671-287c-44be-82fe-3ccfd91ead91-kube-api-access-tbxgj\") pod \"maas-api-e2e-ipp-b-527142-79964f554c-snzzm\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:58.920365 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:58.920302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:45:59.198099 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:45:59.198005    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"]
Sep 15 09:45:59.201857 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:45:59.201819    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d5e671_287c_44be_82fe_3ccfd91ead91.slice/crio-79df477e0e244441aa270d621f4b4789cb0273361876c255ee03e53f02faebea WatchSource:0}: Error finding container 79df477e0e244441aa270d621f4b4789cb0273361876c255ee03e53f02faebea: Status 404 returned error can't find the container with id 79df477e0e244441aa270d621f4b4789cb0273361876c255ee03e53f02faebea
Sep 15 09:46:00.157112 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:00.157072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm" event={"ID":"04d5e671-287c-44be-82fe-3ccfd91ead91","Type":"ContainerStarted","Data":"79df477e0e244441aa270d621f4b4789cb0273361876c255ee03e53f02faebea"}
Sep 15 09:46:01.164097 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:01.164059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm" event={"ID":"04d5e671-287c-44be-82fe-3ccfd91ead91","Type":"ContainerStarted","Data":"6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62"}
Sep 15 09:46:01.164532 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:01.164153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:46:01.182004 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:01.181956    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm" podStartSLOduration=1.888614128 podStartE2EDuration="3.181940514s" totalImagesPullingTime="1.293326386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:46:01.180001393 +0000 UTC m=+1925.009793297" watchObservedRunningTime="2026-09-15 09:46:01.181940514 +0000 UTC m=+1925.011732416"
Sep 15 09:46:03.903371 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:03.903314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:46:07.175281 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:07.175245    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:46:53.701211 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.701176    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"]
Sep 15 09:46:53.705009 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.704988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.707827 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.707807    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6593fe\"/\"kube-root-ca.crt\""
Sep 15 09:46:53.707827 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.707817    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6593fe\"/\"default-dockercfg-c5h57\""
Sep 15 09:46:53.708043 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.707809    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6593fe\"/\"ipp-route-6593fe-kserve-self-signed-certs\""
Sep 15 09:46:53.708043 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.707808    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6593fe\"/\"openshift-service-ca.crt\""
Sep 15 09:46:53.714641 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.714612    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"]
Sep 15 09:46:53.851021 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.850991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-dshm\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.851181 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.851061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-tmp-dir\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.851181 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.851123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cebf51c4-6ad0-42bc-a0bb-566656724e00-tls-certs\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.851181 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.851149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-model-cache\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.851305 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.851219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hxvc\" (UniqueName: \"kubernetes.io/projected/cebf51c4-6ad0-42bc-a0bb-566656724e00-kube-api-access-8hxvc\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.851305 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.851291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-home\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.952181 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.952103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-home\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.952181 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.952145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-dshm\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.952439 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.952183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-tmp-dir\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.952439 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.952226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cebf51c4-6ad0-42bc-a0bb-566656724e00-tls-certs\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.952439 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.952253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-model-cache\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.952439 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.952306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hxvc\" (UniqueName: \"kubernetes.io/projected/cebf51c4-6ad0-42bc-a0bb-566656724e00-kube-api-access-8hxvc\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.952646 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.952540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-home\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.952646 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.952560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-tmp-dir\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.952734 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.952710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-model-cache\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.954420 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.954396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-dshm\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.954736 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.954717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cebf51c4-6ad0-42bc-a0bb-566656724e00-tls-certs\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:53.959405 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:53.959385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hxvc\" (UniqueName: \"kubernetes.io/projected/cebf51c4-6ad0-42bc-a0bb-566656724e00-kube-api-access-8hxvc\") pod \"ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") " pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:54.015956 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:54.015931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:46:54.287783 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:54.287756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"]
Sep 15 09:46:54.290413 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:46:54.290385    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcebf51c4_6ad0_42bc_a0bb_566656724e00.slice/crio-3893d08e479307d4f67e7f0d1e1c004bfb720f117c981e51749573808d674d14 WatchSource:0}: Error finding container 3893d08e479307d4f67e7f0d1e1c004bfb720f117c981e51749573808d674d14: Status 404 returned error can't find the container with id 3893d08e479307d4f67e7f0d1e1c004bfb720f117c981e51749573808d674d14
Sep 15 09:46:54.430291 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:54.430254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q" event={"ID":"cebf51c4-6ad0-42bc-a0bb-566656724e00","Type":"ContainerStarted","Data":"b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11"}
Sep 15 09:46:54.430496 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:54.430325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q" event={"ID":"cebf51c4-6ad0-42bc-a0bb-566656724e00","Type":"ContainerStarted","Data":"3893d08e479307d4f67e7f0d1e1c004bfb720f117c981e51749573808d674d14"}
Sep 15 09:46:54.466467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:54.466423    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q" podStartSLOduration=1.466408009 podStartE2EDuration="1.466408009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:46:54.464930733 +0000 UTC m=+1978.294722636" watchObservedRunningTime="2026-09-15 09:46:54.466408009 +0000 UTC m=+1978.296199912"
Sep 15 09:46:57.600798 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:46:57.600765    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:47:04.016929 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:04.016886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:47:04.017463 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:04.016947    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:47:04.029508 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:04.029476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:47:04.489616 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:04.489587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:47:21.693785 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:21.693743    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:47:35.095149 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.095113    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"]
Sep 15 09:47:35.098882 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.098858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.100823 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.100787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-29a3ae-serving-cert\""
Sep 15 09:47:35.109941 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.109907    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"]
Sep 15 09:47:35.221816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.221785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7799b07c-ae56-44d6-8e0a-8307e1c131a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.221986 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.221829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.221986 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.221890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.221986 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.221937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjcl\" (UniqueName: \"kubernetes.io/projected/7799b07c-ae56-44d6-8e0a-8307e1c131a2-kube-api-access-7kjcl\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.323246 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.323214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjcl\" (UniqueName: \"kubernetes.io/projected/7799b07c-ae56-44d6-8e0a-8307e1c131a2-kube-api-access-7kjcl\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.323455 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.323266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7799b07c-ae56-44d6-8e0a-8307e1c131a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.323455 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.323299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.323455 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.323363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.323979 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.323952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7799b07c-ae56-44d6-8e0a-8307e1c131a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.325803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.325777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.325905 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.325820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.330765 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.330746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjcl\" (UniqueName: \"kubernetes.io/projected/7799b07c-ae56-44d6-8e0a-8307e1c131a2-kube-api-access-7kjcl\") pod \"maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.411383 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.411366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:35.682645 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:35.682620    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"]
Sep 15 09:47:35.685000 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:47:35.684973    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7799b07c_ae56_44d6_8e0a_8307e1c131a2.slice/crio-accc036d73c91d42ea93541eb57190871c024a87cfff1e944027cc4be30de30f WatchSource:0}: Error finding container accc036d73c91d42ea93541eb57190871c024a87cfff1e944027cc4be30de30f: Status 404 returned error can't find the container with id accc036d73c91d42ea93541eb57190871c024a87cfff1e944027cc4be30de30f
Sep 15 09:47:36.637103 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:36.636990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs" event={"ID":"7799b07c-ae56-44d6-8e0a-8307e1c131a2","Type":"ContainerStarted","Data":"f3f3c6b7766a14e37de8ab0ac122e567f33bdeef3a13cdaf91c9cd1e8152b15d"}
Sep 15 09:47:36.637103 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:36.637089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs" event={"ID":"7799b07c-ae56-44d6-8e0a-8307e1c131a2","Type":"ContainerStarted","Data":"accc036d73c91d42ea93541eb57190871c024a87cfff1e944027cc4be30de30f"}
Sep 15 09:47:36.637633 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:36.637115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:47:36.655161 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:36.655102    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs" podStartSLOduration=1.01384735 podStartE2EDuration="1.655083623s" totalImagesPullingTime="641.236273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:47:36.653974074 +0000 UTC m=+2020.483765977" watchObservedRunningTime="2026-09-15 09:47:36.655083623 +0000 UTC m=+2020.484875525"
Sep 15 09:47:41.497638 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:41.497604    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:47:42.647437 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:47:42.647411    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:48:13.619860 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.619828    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"]
Sep 15 09:48:13.620421 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.620022    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs" podUID="7799b07c-ae56-44d6-8e0a-8307e1c131a2" containerName="maas-api" containerID="cri-o://f3f3c6b7766a14e37de8ab0ac122e567f33bdeef3a13cdaf91c9cd1e8152b15d" gracePeriod=30
Sep 15 09:48:13.814413 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.814372    2579 generic.go:356] "Generic (PLEG): container finished" podID="7799b07c-ae56-44d6-8e0a-8307e1c131a2" containerID="f3f3c6b7766a14e37de8ab0ac122e567f33bdeef3a13cdaf91c9cd1e8152b15d" exitCode=0
Sep 15 09:48:13.814590 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.814428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs" event={"ID":"7799b07c-ae56-44d6-8e0a-8307e1c131a2","Type":"ContainerDied","Data":"f3f3c6b7766a14e37de8ab0ac122e567f33bdeef3a13cdaf91c9cd1e8152b15d"}
Sep 15 09:48:13.866883 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.866857    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:48:13.964208 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.964177    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-maas-api-tls\") pod \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") "
Sep 15 09:48:13.964394 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.964244    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7799b07c-ae56-44d6-8e0a-8307e1c131a2-openshift-service-ca\") pod \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") "
Sep 15 09:48:13.964394 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.964299    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-metrics-certs\") pod \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") "
Sep 15 09:48:13.964394 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.964374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kjcl\" (UniqueName: \"kubernetes.io/projected/7799b07c-ae56-44d6-8e0a-8307e1c131a2-kube-api-access-7kjcl\") pod \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\" (UID: \"7799b07c-ae56-44d6-8e0a-8307e1c131a2\") "
Sep 15 09:48:13.964659 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.964623    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7799b07c-ae56-44d6-8e0a-8307e1c131a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7799b07c-ae56-44d6-8e0a-8307e1c131a2" (UID: "7799b07c-ae56-44d6-8e0a-8307e1c131a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:48:13.966517 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.966495    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7799b07c-ae56-44d6-8e0a-8307e1c131a2" (UID: "7799b07c-ae56-44d6-8e0a-8307e1c131a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:48:13.966598 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.966519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7799b07c-ae56-44d6-8e0a-8307e1c131a2" (UID: "7799b07c-ae56-44d6-8e0a-8307e1c131a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:48:13.966640 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:13.966587    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7799b07c-ae56-44d6-8e0a-8307e1c131a2-kube-api-access-7kjcl" (OuterVolumeSpecName: "kube-api-access-7kjcl") pod "7799b07c-ae56-44d6-8e0a-8307e1c131a2" (UID: "7799b07c-ae56-44d6-8e0a-8307e1c131a2"). InnerVolumeSpecName "kube-api-access-7kjcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:48:14.065673 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:14.065645    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:48:14.065673 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:14.065669    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7799b07c-ae56-44d6-8e0a-8307e1c131a2-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:48:14.065673 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:14.065679    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7799b07c-ae56-44d6-8e0a-8307e1c131a2-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:48:14.065871 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:14.065688    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjcl\" (UniqueName: \"kubernetes.io/projected/7799b07c-ae56-44d6-8e0a-8307e1c131a2-kube-api-access-7kjcl\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:48:14.820795 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:14.820771    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"
Sep 15 09:48:14.820795 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:14.820781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs" event={"ID":"7799b07c-ae56-44d6-8e0a-8307e1c131a2","Type":"ContainerDied","Data":"accc036d73c91d42ea93541eb57190871c024a87cfff1e944027cc4be30de30f"}
Sep 15 09:48:14.821230 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:14.820835    2579 scope.go:117] "RemoveContainer" containerID="f3f3c6b7766a14e37de8ab0ac122e567f33bdeef3a13cdaf91c9cd1e8152b15d"
Sep 15 09:48:14.846050 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:14.846023    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"]
Sep 15 09:48:14.849904 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:14.849882    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-29a3ae-7956f88c75-57zgs"]
Sep 15 09:48:16.674991 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:16.674958    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7799b07c-ae56-44d6-8e0a-8307e1c131a2" path="/var/lib/kubelet/pods/7799b07c-ae56-44d6-8e0a-8307e1c131a2/volumes"
Sep 15 09:48:18.058269 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:18.058237    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:48:39.517924 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:39.517850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:48:42.023298 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.023258    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"]
Sep 15 09:48:42.023714 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.023465    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v" podUID="c5303e6f-b116-4845-8411-3cdf0f7a89f9" containerName="maas-api" containerID="cri-o://783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7" gracePeriod=30
Sep 15 09:48:42.270536 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.270508    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:48:42.311058 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.310987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jqr9\" (UniqueName: \"kubernetes.io/projected/c5303e6f-b116-4845-8411-3cdf0f7a89f9-kube-api-access-6jqr9\") pod \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") "
Sep 15 09:48:42.311194 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.311066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5303e6f-b116-4845-8411-3cdf0f7a89f9-openshift-service-ca\") pod \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") "
Sep 15 09:48:42.311194 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.311132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-metrics-certs\") pod \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") "
Sep 15 09:48:42.311194 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.311168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-maas-api-tls\") pod \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\" (UID: \"c5303e6f-b116-4845-8411-3cdf0f7a89f9\") "
Sep 15 09:48:42.311484 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.311457    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5303e6f-b116-4845-8411-3cdf0f7a89f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5303e6f-b116-4845-8411-3cdf0f7a89f9" (UID: "c5303e6f-b116-4845-8411-3cdf0f7a89f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:48:42.313214 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.313185    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5303e6f-b116-4845-8411-3cdf0f7a89f9" (UID: "c5303e6f-b116-4845-8411-3cdf0f7a89f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:48:42.313214 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.313184    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5303e6f-b116-4845-8411-3cdf0f7a89f9-kube-api-access-6jqr9" (OuterVolumeSpecName: "kube-api-access-6jqr9") pod "c5303e6f-b116-4845-8411-3cdf0f7a89f9" (UID: "c5303e6f-b116-4845-8411-3cdf0f7a89f9"). InnerVolumeSpecName "kube-api-access-6jqr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:48:42.313428 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.313232    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c5303e6f-b116-4845-8411-3cdf0f7a89f9" (UID: "c5303e6f-b116-4845-8411-3cdf0f7a89f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:48:42.412565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.412540    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jqr9\" (UniqueName: \"kubernetes.io/projected/c5303e6f-b116-4845-8411-3cdf0f7a89f9-kube-api-access-6jqr9\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:48:42.412565 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.412563    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5303e6f-b116-4845-8411-3cdf0f7a89f9-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:48:42.412712 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.412574    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:48:42.412712 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.412584    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5303e6f-b116-4845-8411-3cdf0f7a89f9-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:48:42.960750 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.960673    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5303e6f-b116-4845-8411-3cdf0f7a89f9" containerID="783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7" exitCode=0
Sep 15 09:48:42.960750 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.960718    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"
Sep 15 09:48:42.960970 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.960760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v" event={"ID":"c5303e6f-b116-4845-8411-3cdf0f7a89f9","Type":"ContainerDied","Data":"783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7"}
Sep 15 09:48:42.960970 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.960816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v" event={"ID":"c5303e6f-b116-4845-8411-3cdf0f7a89f9","Type":"ContainerDied","Data":"e71b1d46616fae6921e86d0b955edacd23e78aa808c0f6bab936c5427c1add81"}
Sep 15 09:48:42.960970 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.960844    2579 scope.go:117] "RemoveContainer" containerID="783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7"
Sep 15 09:48:42.969202 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.969187    2579 scope.go:117] "RemoveContainer" containerID="783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7"
Sep 15 09:48:42.969484 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:48:42.969468    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7\": container with ID starting with 783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7 not found: ID does not exist" containerID="783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7"
Sep 15 09:48:42.969536 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.969494    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7"} err="failed to get container status \"783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7\": rpc error: code = NotFound desc = could not find container \"783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7\": container with ID starting with 783421f1e42031f62342f04283315076e52a19be70905781f4d4f28169d7d6c7 not found: ID does not exist"
Sep 15 09:48:42.980317 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.980296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"]
Sep 15 09:48:42.984706 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:42.984683    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6593fe-64bd67b857-tjz5v"]
Sep 15 09:48:44.675528 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:44.675498    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5303e6f-b116-4845-8411-3cdf0f7a89f9" path="/var/lib/kubelet/pods/c5303e6f-b116-4845-8411-3cdf0f7a89f9/volumes"
Sep 15 09:48:47.702345 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:47.702292    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"]
Sep 15 09:48:52.755788 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:48:52.755755    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:49:02.721944 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:02.721888    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q" podUID="cebf51c4-6ad0-42bc-a0bb-566656724e00" containerName="main" containerID="cri-o://b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11" gracePeriod=45
Sep 15 09:49:02.970668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:02.970645    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:49:03.060384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.060280    2579 generic.go:356] "Generic (PLEG): container finished" podID="cebf51c4-6ad0-42bc-a0bb-566656724e00" containerID="b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11" exitCode=0
Sep 15 09:49:03.060384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.060323    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"
Sep 15 09:49:03.060384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.060366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q" event={"ID":"cebf51c4-6ad0-42bc-a0bb-566656724e00","Type":"ContainerDied","Data":"b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11"}
Sep 15 09:49:03.060656 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.060409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q" event={"ID":"cebf51c4-6ad0-42bc-a0bb-566656724e00","Type":"ContainerDied","Data":"3893d08e479307d4f67e7f0d1e1c004bfb720f117c981e51749573808d674d14"}
Sep 15 09:49:03.060656 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.060428    2579 scope.go:117] "RemoveContainer" containerID="b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11"
Sep 15 09:49:03.068478 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.068465    2579 scope.go:117] "RemoveContainer" containerID="b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11"
Sep 15 09:49:03.068731 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:49:03.068713    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11\": container with ID starting with b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11 not found: ID does not exist" containerID="b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11"
Sep 15 09:49:03.068777 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.068738    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11"} err="failed to get container status \"b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11\": rpc error: code = NotFound desc = could not find container \"b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11\": container with ID starting with b3423604ccb1e6bec92232a239ecb0c46028432d8d3452eaa7846c2ef9142e11 not found: ID does not exist"
Sep 15 09:49:03.090072 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090045    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-model-cache\") pod \"cebf51c4-6ad0-42bc-a0bb-566656724e00\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") "
Sep 15 09:49:03.090182 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090080    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-tmp-dir\") pod \"cebf51c4-6ad0-42bc-a0bb-566656724e00\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") "
Sep 15 09:49:03.090182 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090120    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cebf51c4-6ad0-42bc-a0bb-566656724e00-tls-certs\") pod \"cebf51c4-6ad0-42bc-a0bb-566656724e00\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") "
Sep 15 09:49:03.090182 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090145    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hxvc\" (UniqueName: \"kubernetes.io/projected/cebf51c4-6ad0-42bc-a0bb-566656724e00-kube-api-access-8hxvc\") pod \"cebf51c4-6ad0-42bc-a0bb-566656724e00\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") "
Sep 15 09:49:03.090368 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090204    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-home\") pod \"cebf51c4-6ad0-42bc-a0bb-566656724e00\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") "
Sep 15 09:49:03.090368 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-dshm\") pod \"cebf51c4-6ad0-42bc-a0bb-566656724e00\" (UID: \"cebf51c4-6ad0-42bc-a0bb-566656724e00\") "
Sep 15 09:49:03.090368 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-model-cache" (OuterVolumeSpecName: "model-cache") pod "cebf51c4-6ad0-42bc-a0bb-566656724e00" (UID: "cebf51c4-6ad0-42bc-a0bb-566656724e00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:49:03.090505 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090386    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cebf51c4-6ad0-42bc-a0bb-566656724e00" (UID: "cebf51c4-6ad0-42bc-a0bb-566656724e00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:49:03.090545 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090522    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-home" (OuterVolumeSpecName: "home") pod "cebf51c4-6ad0-42bc-a0bb-566656724e00" (UID: "cebf51c4-6ad0-42bc-a0bb-566656724e00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:49:03.090665 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090648    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:03.090726 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090684    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:03.090726 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.090693    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:03.092260 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.092234    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-dshm" (OuterVolumeSpecName: "dshm") pod "cebf51c4-6ad0-42bc-a0bb-566656724e00" (UID: "cebf51c4-6ad0-42bc-a0bb-566656724e00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:49:03.092382 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.092360    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cebf51c4-6ad0-42bc-a0bb-566656724e00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cebf51c4-6ad0-42bc-a0bb-566656724e00" (UID: "cebf51c4-6ad0-42bc-a0bb-566656724e00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:49:03.092448 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.092364    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebf51c4-6ad0-42bc-a0bb-566656724e00-kube-api-access-8hxvc" (OuterVolumeSpecName: "kube-api-access-8hxvc") pod "cebf51c4-6ad0-42bc-a0bb-566656724e00" (UID: "cebf51c4-6ad0-42bc-a0bb-566656724e00"). InnerVolumeSpecName "kube-api-access-8hxvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:49:03.191883 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.191854    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cebf51c4-6ad0-42bc-a0bb-566656724e00-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:03.191883 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.191882    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cebf51c4-6ad0-42bc-a0bb-566656724e00-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:03.192004 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.191892    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hxvc\" (UniqueName: \"kubernetes.io/projected/cebf51c4-6ad0-42bc-a0bb-566656724e00-kube-api-access-8hxvc\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:03.380756 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.380735    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"]
Sep 15 09:49:03.384991 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:03.384970    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6593fe/ipp-route-6593fe-kserve-6d4fbbcc87-j7k7q"]
Sep 15 09:49:04.675497 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:04.675464    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cebf51c4-6ad0-42bc-a0bb-566656724e00" path="/var/lib/kubelet/pods/cebf51c4-6ad0-42bc-a0bb-566656724e00/volumes"
Sep 15 09:49:41.775129 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:41.775096    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"]
Sep 15 09:49:42.247731 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.247669    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm" podUID="04d5e671-287c-44be-82fe-3ccfd91ead91" containerName="maas-api" containerID="cri-o://6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62" gracePeriod=30
Sep 15 09:49:42.500251 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.500198    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:49:42.528697 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.528666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-maas-api-tls\") pod \"04d5e671-287c-44be-82fe-3ccfd91ead91\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") "
Sep 15 09:49:42.528857 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.528745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d5e671-287c-44be-82fe-3ccfd91ead91-openshift-service-ca\") pod \"04d5e671-287c-44be-82fe-3ccfd91ead91\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") "
Sep 15 09:49:42.528857 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.528850    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-metrics-certs\") pod \"04d5e671-287c-44be-82fe-3ccfd91ead91\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") "
Sep 15 09:49:42.531555 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.528885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbxgj\" (UniqueName: \"kubernetes.io/projected/04d5e671-287c-44be-82fe-3ccfd91ead91-kube-api-access-tbxgj\") pod \"04d5e671-287c-44be-82fe-3ccfd91ead91\" (UID: \"04d5e671-287c-44be-82fe-3ccfd91ead91\") "
Sep 15 09:49:42.531555 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.529367    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d5e671-287c-44be-82fe-3ccfd91ead91-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04d5e671-287c-44be-82fe-3ccfd91ead91" (UID: "04d5e671-287c-44be-82fe-3ccfd91ead91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:49:42.531555 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.530794    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d5e671-287c-44be-82fe-3ccfd91ead91-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:42.536356 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.532209    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "04d5e671-287c-44be-82fe-3ccfd91ead91" (UID: "04d5e671-287c-44be-82fe-3ccfd91ead91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:49:42.537963 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.537599    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04d5e671-287c-44be-82fe-3ccfd91ead91" (UID: "04d5e671-287c-44be-82fe-3ccfd91ead91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:49:42.539787 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.539763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d5e671-287c-44be-82fe-3ccfd91ead91-kube-api-access-tbxgj" (OuterVolumeSpecName: "kube-api-access-tbxgj") pod "04d5e671-287c-44be-82fe-3ccfd91ead91" (UID: "04d5e671-287c-44be-82fe-3ccfd91ead91"). InnerVolumeSpecName "kube-api-access-tbxgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:49:42.631467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.631445    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:42.631467 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.631465    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d5e671-287c-44be-82fe-3ccfd91ead91-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:42.631602 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:42.631474    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxgj\" (UniqueName: \"kubernetes.io/projected/04d5e671-287c-44be-82fe-3ccfd91ead91-kube-api-access-tbxgj\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:49:43.253487 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:43.253450    2579 generic.go:356] "Generic (PLEG): container finished" podID="04d5e671-287c-44be-82fe-3ccfd91ead91" containerID="6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62" exitCode=0
Sep 15 09:49:43.253896 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:43.253507    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"
Sep 15 09:49:43.253896 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:43.253537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm" event={"ID":"04d5e671-287c-44be-82fe-3ccfd91ead91","Type":"ContainerDied","Data":"6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62"}
Sep 15 09:49:43.253896 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:43.253590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm" event={"ID":"04d5e671-287c-44be-82fe-3ccfd91ead91","Type":"ContainerDied","Data":"79df477e0e244441aa270d621f4b4789cb0273361876c255ee03e53f02faebea"}
Sep 15 09:49:43.253896 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:43.253612    2579 scope.go:117] "RemoveContainer" containerID="6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62"
Sep 15 09:49:43.262525 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:43.262507    2579 scope.go:117] "RemoveContainer" containerID="6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62"
Sep 15 09:49:43.262793 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:49:43.262773    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62\": container with ID starting with 6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62 not found: ID does not exist" containerID="6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62"
Sep 15 09:49:43.262846 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:43.262802    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62"} err="failed to get container status \"6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62\": rpc error: code = NotFound desc = could not find container \"6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62\": container with ID starting with 6ccc01379368cd98a207d960b789ef3ac3facbc42ec958e23a0b089172e43b62 not found: ID does not exist"
Sep 15 09:49:43.274031 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:43.274007    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"]
Sep 15 09:49:43.276989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:43.276965    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-527142-79964f554c-snzzm"]
Sep 15 09:49:44.674891 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:44.674860    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04d5e671-287c-44be-82fe-3ccfd91ead91" path="/var/lib/kubelet/pods/04d5e671-287c-44be-82fe-3ccfd91ead91/volumes"
Sep 15 09:49:45.103320 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:49:45.103232    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:50:02.733164 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:02.733124    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:50:17.535182 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.535147    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"]
Sep 15 09:50:17.538461 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.535363    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp" podUID="a00ca908-2ddd-447b-bf38-6e010f11c279" containerName="maas-api" containerID="cri-o://f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946" gracePeriod=30
Sep 15 09:50:17.784564 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.784539    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:50:17.927085 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.927059    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-metrics-certs\") pod \"a00ca908-2ddd-447b-bf38-6e010f11c279\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") "
Sep 15 09:50:17.927266 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.927160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ca908-2ddd-447b-bf38-6e010f11c279-openshift-service-ca\") pod \"a00ca908-2ddd-447b-bf38-6e010f11c279\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") "
Sep 15 09:50:17.927266 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.927206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65zc7\" (UniqueName: \"kubernetes.io/projected/a00ca908-2ddd-447b-bf38-6e010f11c279-kube-api-access-65zc7\") pod \"a00ca908-2ddd-447b-bf38-6e010f11c279\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") "
Sep 15 09:50:17.927266 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.927242    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-maas-api-tls\") pod \"a00ca908-2ddd-447b-bf38-6e010f11c279\" (UID: \"a00ca908-2ddd-447b-bf38-6e010f11c279\") "
Sep 15 09:50:17.927534 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.927506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00ca908-2ddd-447b-bf38-6e010f11c279-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a00ca908-2ddd-447b-bf38-6e010f11c279" (UID: "a00ca908-2ddd-447b-bf38-6e010f11c279"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:50:17.929141 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.929115    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a00ca908-2ddd-447b-bf38-6e010f11c279" (UID: "a00ca908-2ddd-447b-bf38-6e010f11c279"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:50:17.929370 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.929313    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00ca908-2ddd-447b-bf38-6e010f11c279-kube-api-access-65zc7" (OuterVolumeSpecName: "kube-api-access-65zc7") pod "a00ca908-2ddd-447b-bf38-6e010f11c279" (UID: "a00ca908-2ddd-447b-bf38-6e010f11c279"). InnerVolumeSpecName "kube-api-access-65zc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:50:17.929489 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:17.929381    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a00ca908-2ddd-447b-bf38-6e010f11c279" (UID: "a00ca908-2ddd-447b-bf38-6e010f11c279"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:50:18.028410 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.028379    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:50:18.028410 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.028403    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00ca908-2ddd-447b-bf38-6e010f11c279-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:50:18.028410 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.028413    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00ca908-2ddd-447b-bf38-6e010f11c279-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:50:18.028622 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.028422    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65zc7\" (UniqueName: \"kubernetes.io/projected/a00ca908-2ddd-447b-bf38-6e010f11c279-kube-api-access-65zc7\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:50:18.427508 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.427470    2579 generic.go:356] "Generic (PLEG): container finished" podID="a00ca908-2ddd-447b-bf38-6e010f11c279" containerID="f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946" exitCode=0
Sep 15 09:50:18.427669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.427514    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"
Sep 15 09:50:18.427669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.427531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp" event={"ID":"a00ca908-2ddd-447b-bf38-6e010f11c279","Type":"ContainerDied","Data":"f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946"}
Sep 15 09:50:18.427669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.427583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp" event={"ID":"a00ca908-2ddd-447b-bf38-6e010f11c279","Type":"ContainerDied","Data":"dc663957f657d123996748e849fc00806095b82af6b5c3d9fc9c3d44d6214d58"}
Sep 15 09:50:18.427669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.427612    2579 scope.go:117] "RemoveContainer" containerID="f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946"
Sep 15 09:50:18.436627 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.436609    2579 scope.go:117] "RemoveContainer" containerID="f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946"
Sep 15 09:50:18.436882 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:50:18.436864    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946\": container with ID starting with f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946 not found: ID does not exist" containerID="f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946"
Sep 15 09:50:18.436950 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.436889    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946"} err="failed to get container status \"f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946\": rpc error: code = NotFound desc = could not find container \"f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946\": container with ID starting with f9c7540801b3378e9b5d0ed8f8be426c814bd8d7817b1e707a920a084bfd2946 not found: ID does not exist"
Sep 15 09:50:18.447597 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.447573    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"]
Sep 15 09:50:18.449669 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.449647    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b283ee-7c8c64cfd-shxdp"]
Sep 15 09:50:18.676686 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:18.676657    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a00ca908-2ddd-447b-bf38-6e010f11c279" path="/var/lib/kubelet/pods/a00ca908-2ddd-447b-bf38-6e010f11c279/volumes"
Sep 15 09:50:20.606518 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:20.606481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:50:29.095837 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:50:29.095800    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:51:52.093655 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:51:52.093621    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:52:18.393160 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:52:18.393131    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:52:26.000861 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:52:26.000825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:52:32.995827 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:52:32.995798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:52:39.802017 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:52:39.801972    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:52:47.000220 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:52:47.000185    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:52:57.897810 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:52:57.897772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:53:05.896634 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:53:05.896596    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:54:35.692174 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:54:35.692098    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:54:43.611109 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:54:43.611073    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:54:49.598412 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:54:49.598377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:54:58.093937 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:54:58.093906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:55:05.408494 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.408446    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"]
Sep 15 09:55:05.408996 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.408981    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5303e6f-b116-4845-8411-3cdf0f7a89f9" containerName="maas-api"
Sep 15 09:55:05.409045 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409000    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5303e6f-b116-4845-8411-3cdf0f7a89f9" containerName="maas-api"
Sep 15 09:55:05.409045 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409015    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d5e671-287c-44be-82fe-3ccfd91ead91" containerName="maas-api"
Sep 15 09:55:05.409045 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409025    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d5e671-287c-44be-82fe-3ccfd91ead91" containerName="maas-api"
Sep 15 09:55:05.409045 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409036    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7799b07c-ae56-44d6-8e0a-8307e1c131a2" containerName="maas-api"
Sep 15 09:55:05.409045 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409044    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7799b07c-ae56-44d6-8e0a-8307e1c131a2" containerName="maas-api"
Sep 15 09:55:05.409200 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409064    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebf51c4-6ad0-42bc-a0bb-566656724e00" containerName="main"
Sep 15 09:55:05.409200 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409072    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebf51c4-6ad0-42bc-a0bb-566656724e00" containerName="main"
Sep 15 09:55:05.409200 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409087    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a00ca908-2ddd-447b-bf38-6e010f11c279" containerName="maas-api"
Sep 15 09:55:05.409200 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409095    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a00ca908-2ddd-447b-bf38-6e010f11c279" containerName="maas-api"
Sep 15 09:55:05.409200 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409178    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5303e6f-b116-4845-8411-3cdf0f7a89f9" containerName="maas-api"
Sep 15 09:55:05.409200 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409189    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7799b07c-ae56-44d6-8e0a-8307e1c131a2" containerName="maas-api"
Sep 15 09:55:05.409200 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409198    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a00ca908-2ddd-447b-bf38-6e010f11c279" containerName="maas-api"
Sep 15 09:55:05.409431 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409208    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d5e671-287c-44be-82fe-3ccfd91ead91" containerName="maas-api"
Sep 15 09:55:05.409431 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.409218    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebf51c4-6ad0-42bc-a0bb-566656724e00" containerName="main"
Sep 15 09:55:05.412512 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.412486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.414622 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.414599    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-52753c93-kserve-self-signed-certs\""
Sep 15 09:55:05.419104 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.419081    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"]
Sep 15 09:55:05.549901 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.549869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tmp-dir\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.549901 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.549907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tls-certs\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.550106 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.549977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-dshm\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.550106 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.550040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-model-cache\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.550106 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.550059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-home\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.550230 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.550101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pm8\" (UniqueName: \"kubernetes.io/projected/5b819c43-e9f7-47ae-b555-7f4dbb823dea-kube-api-access-g5pm8\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.651172 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.651143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tmp-dir\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.651306 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.651182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tls-certs\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.651306 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.651244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-dshm\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.651306 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.651282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-model-cache\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.651520 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.651309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-home\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.651520 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.651371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pm8\" (UniqueName: \"kubernetes.io/projected/5b819c43-e9f7-47ae-b555-7f4dbb823dea-kube-api-access-g5pm8\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.651639 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.651581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tmp-dir\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.651639 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.651607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-model-cache\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.651733 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.651714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-home\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.653532 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.653507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-dshm\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.653779 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.653762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tls-certs\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.658959 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.658913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pm8\" (UniqueName: \"kubernetes.io/projected/5b819c43-e9f7-47ae-b555-7f4dbb823dea-kube-api-access-g5pm8\") pod \"e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") " pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.722685 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.722661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:05.997755 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:05.997684    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"]
Sep 15 09:55:06.000074 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:55:06.000044    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b819c43_e9f7_47ae_b555_7f4dbb823dea.slice/crio-9e262768b50bd686b5818b7e57c0f9ff909e0c1652b612c931ef3da0a385048d WatchSource:0}: Error finding container 9e262768b50bd686b5818b7e57c0f9ff909e0c1652b612c931ef3da0a385048d: Status 404 returned error can't find the container with id 9e262768b50bd686b5818b7e57c0f9ff909e0c1652b612c931ef3da0a385048d
Sep 15 09:55:06.773256 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:06.773218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg" event={"ID":"5b819c43-e9f7-47ae-b555-7f4dbb823dea","Type":"ContainerStarted","Data":"bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905"}
Sep 15 09:55:06.773665 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:06.773265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg" event={"ID":"5b819c43-e9f7-47ae-b555-7f4dbb823dea","Type":"ContainerStarted","Data":"9e262768b50bd686b5818b7e57c0f9ff909e0c1652b612c931ef3da0a385048d"}
Sep 15 09:55:06.794279 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:06.794230    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg" podStartSLOduration=1.794208145 podStartE2EDuration="1.794208145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:55:06.793475981 +0000 UTC m=+2470.623267885" watchObservedRunningTime="2026-09-15 09:55:06.794208145 +0000 UTC m=+2470.624000049"
Sep 15 09:55:08.402236 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:08.402201    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:55:15.723303 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:15.723262    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:15.723303 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:15.723309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:15.735802 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:15.735773    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:15.824620 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:15.824588    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:34.909209 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:34.909174    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"]
Sep 15 09:55:35.297428 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:35.297326    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:55:43.795472 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:43.795438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:55:49.928938 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:49.928827    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg" podUID="5b819c43-e9f7-47ae-b555-7f4dbb823dea" containerName="main" containerID="cri-o://bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905" gracePeriod=45
Sep 15 09:55:50.166509 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.166487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:50.225142 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225081    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tmp-dir\") pod \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") "
Sep 15 09:55:50.225142 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225125    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-model-cache\") pod \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") "
Sep 15 09:55:50.225366 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225149    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5pm8\" (UniqueName: \"kubernetes.io/projected/5b819c43-e9f7-47ae-b555-7f4dbb823dea-kube-api-access-g5pm8\") pod \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") "
Sep 15 09:55:50.225366 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-home\") pod \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") "
Sep 15 09:55:50.225366 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-dshm\") pod \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") "
Sep 15 09:55:50.225366 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tls-certs\") pod \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\" (UID: \"5b819c43-e9f7-47ae-b555-7f4dbb823dea\") "
Sep 15 09:55:50.225544 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b819c43-e9f7-47ae-b555-7f4dbb823dea" (UID: "5b819c43-e9f7-47ae-b555-7f4dbb823dea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:55:50.225544 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225405    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b819c43-e9f7-47ae-b555-7f4dbb823dea" (UID: "5b819c43-e9f7-47ae-b555-7f4dbb823dea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:55:50.225544 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225440    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-home" (OuterVolumeSpecName: "home") pod "5b819c43-e9f7-47ae-b555-7f4dbb823dea" (UID: "5b819c43-e9f7-47ae-b555-7f4dbb823dea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:55:50.225544 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225529    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:55:50.225544 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225540    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:55:50.225702 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.225549    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:55:50.227514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.227487    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b819c43-e9f7-47ae-b555-7f4dbb823dea-kube-api-access-g5pm8" (OuterVolumeSpecName: "kube-api-access-g5pm8") pod "5b819c43-e9f7-47ae-b555-7f4dbb823dea" (UID: "5b819c43-e9f7-47ae-b555-7f4dbb823dea"). InnerVolumeSpecName "kube-api-access-g5pm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:55:50.227621 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.227516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b819c43-e9f7-47ae-b555-7f4dbb823dea" (UID: "5b819c43-e9f7-47ae-b555-7f4dbb823dea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:55:50.227621 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.227603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-dshm" (OuterVolumeSpecName: "dshm") pod "5b819c43-e9f7-47ae-b555-7f4dbb823dea" (UID: "5b819c43-e9f7-47ae-b555-7f4dbb823dea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:55:50.326773 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.326743    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5pm8\" (UniqueName: \"kubernetes.io/projected/5b819c43-e9f7-47ae-b555-7f4dbb823dea-kube-api-access-g5pm8\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:55:50.326773 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.326768    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b819c43-e9f7-47ae-b555-7f4dbb823dea-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:55:50.326912 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.326783    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b819c43-e9f7-47ae-b555-7f4dbb823dea-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:55:50.979605 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.979524    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b819c43-e9f7-47ae-b555-7f4dbb823dea" containerID="bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905" exitCode=0
Sep 15 09:55:50.979605 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.979567    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"
Sep 15 09:55:50.979605 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.979577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg" event={"ID":"5b819c43-e9f7-47ae-b555-7f4dbb823dea","Type":"ContainerDied","Data":"bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905"}
Sep 15 09:55:50.980324 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.979618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg" event={"ID":"5b819c43-e9f7-47ae-b555-7f4dbb823dea","Type":"ContainerDied","Data":"9e262768b50bd686b5818b7e57c0f9ff909e0c1652b612c931ef3da0a385048d"}
Sep 15 09:55:50.980324 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.979636    2579 scope.go:117] "RemoveContainer" containerID="bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905"
Sep 15 09:55:50.988258 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.988237    2579 scope.go:117] "RemoveContainer" containerID="bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905"
Sep 15 09:55:50.988567 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:55:50.988549    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905\": container with ID starting with bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905 not found: ID does not exist" containerID="bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905"
Sep 15 09:55:50.988650 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.988576    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905"} err="failed to get container status \"bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905\": rpc error: code = NotFound desc = could not find container \"bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905\": container with ID starting with bde385411721064f97a79342b004dc1035004efd14494a0fdaa7637d69ddf905 not found: ID does not exist"
Sep 15 09:55:50.993959 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.993933    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"]
Sep 15 09:55:50.995712 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.995691    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-52753c93-kserve-6cf7bcdb55-xhfxg"]
Sep 15 09:55:50.998258 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:50.998239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:55:52.675989 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:52.675951    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b819c43-e9f7-47ae-b555-7f4dbb823dea" path="/var/lib/kubelet/pods/5b819c43-e9f7-47ae-b555-7f4dbb823dea/volumes"
Sep 15 09:55:57.905005 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:57.904966    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:55:57.977598 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:57.977559    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"]
Sep 15 09:55:57.977964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:57.977947    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b819c43-e9f7-47ae-b555-7f4dbb823dea" containerName="main"
Sep 15 09:55:57.977964 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:57.977961    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b819c43-e9f7-47ae-b555-7f4dbb823dea" containerName="main"
Sep 15 09:55:57.978131 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:57.978030    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b819c43-e9f7-47ae-b555-7f4dbb823dea" containerName="main"
Sep 15 09:55:57.982616 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:57.982591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:57.994599 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:57.994565    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"]
Sep 15 09:55:58.098955 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.098915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xpv\" (UniqueName: \"kubernetes.io/projected/71e9da57-40b3-4057-9be7-204256ef9acc-kube-api-access-k2xpv\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.099110 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.098992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e9da57-40b3-4057-9be7-204256ef9acc-openshift-service-ca\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.099110 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.099014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e9da57-40b3-4057-9be7-204256ef9acc-metrics-certs\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.099110 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.099065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e9da57-40b3-4057-9be7-204256ef9acc-maas-api-tls\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.200326 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.200291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e9da57-40b3-4057-9be7-204256ef9acc-openshift-service-ca\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.200486 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.200354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e9da57-40b3-4057-9be7-204256ef9acc-metrics-certs\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.200486 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.200425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e9da57-40b3-4057-9be7-204256ef9acc-maas-api-tls\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.200486 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.200476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xpv\" (UniqueName: \"kubernetes.io/projected/71e9da57-40b3-4057-9be7-204256ef9acc-kube-api-access-k2xpv\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.201118 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.201085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e9da57-40b3-4057-9be7-204256ef9acc-openshift-service-ca\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.202996 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.202975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e9da57-40b3-4057-9be7-204256ef9acc-metrics-certs\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.203114 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.203093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e9da57-40b3-4057-9be7-204256ef9acc-maas-api-tls\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.208683 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.208659    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xpv\" (UniqueName: \"kubernetes.io/projected/71e9da57-40b3-4057-9be7-204256ef9acc-kube-api-access-k2xpv\") pod \"maas-api-c4664ccd7-6q5sg\" (UID: \"71e9da57-40b3-4057-9be7-204256ef9acc\") " pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.294849 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.294807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:55:58.575542 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.575469    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"]
Sep 15 09:55:58.578224 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:55:58.578188    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e9da57_40b3_4057_9be7_204256ef9acc.slice/crio-30a0cf3db84b51448de7bf4db2265e70875f0fcde718092264fcaa59f3476e94 WatchSource:0}: Error finding container 30a0cf3db84b51448de7bf4db2265e70875f0fcde718092264fcaa59f3476e94: Status 404 returned error can't find the container with id 30a0cf3db84b51448de7bf4db2265e70875f0fcde718092264fcaa59f3476e94
Sep 15 09:55:58.579597 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:58.579577    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:55:59.021803 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:55:59.021766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg" event={"ID":"71e9da57-40b3-4057-9be7-204256ef9acc","Type":"ContainerStarted","Data":"30a0cf3db84b51448de7bf4db2265e70875f0fcde718092264fcaa59f3476e94"}
Sep 15 09:56:00.027551 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:00.027508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg" event={"ID":"71e9da57-40b3-4057-9be7-204256ef9acc","Type":"ContainerStarted","Data":"2490a2247988eb7934113bfa2343de9109b70a1a2d836585d57628bc6d811929"}
Sep 15 09:56:00.027946 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:00.027673    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:56:00.048212 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:00.048165    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg" podStartSLOduration=2.665399982 podStartE2EDuration="3.048152967s" totalImagesPullingTime="382.752985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:56:00.04730265 +0000 UTC m=+2523.877094556" watchObservedRunningTime="2026-09-15 09:56:00.048152967 +0000 UTC m=+2523.877944900"
Sep 15 09:56:05.697703 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:05.697658    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:56:06.038552 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:06.038479    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c4664ccd7-6q5sg"
Sep 15 09:56:06.087374 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:06.087311    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"]
Sep 15 09:56:06.087611 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:06.087548    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48" podUID="1f0d04e9-a9a4-4913-9350-00ce30b70788" containerName="maas-api" containerID="cri-o://9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f" gracePeriod=30
Sep 15 09:56:07.341362 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.341314    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:56:07.486894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.486800    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n48k\" (UniqueName: \"kubernetes.io/projected/1f0d04e9-a9a4-4913-9350-00ce30b70788-kube-api-access-5n48k\") pod \"1f0d04e9-a9a4-4913-9350-00ce30b70788\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") "
Sep 15 09:56:07.486894 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.486876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d04e9-a9a4-4913-9350-00ce30b70788-openshift-service-ca\") pod \"1f0d04e9-a9a4-4913-9350-00ce30b70788\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") "
Sep 15 09:56:07.487107 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.486904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-maas-api-tls\") pod \"1f0d04e9-a9a4-4913-9350-00ce30b70788\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") "
Sep 15 09:56:07.487107 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.486965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-metrics-certs\") pod \"1f0d04e9-a9a4-4913-9350-00ce30b70788\" (UID: \"1f0d04e9-a9a4-4913-9350-00ce30b70788\") "
Sep 15 09:56:07.487327 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.487298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0d04e9-a9a4-4913-9350-00ce30b70788-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f0d04e9-a9a4-4913-9350-00ce30b70788" (UID: "1f0d04e9-a9a4-4913-9350-00ce30b70788"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:56:07.489106 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.489065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f0d04e9-a9a4-4913-9350-00ce30b70788" (UID: "1f0d04e9-a9a4-4913-9350-00ce30b70788"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:56:07.489201 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.489104    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f0d04e9-a9a4-4913-9350-00ce30b70788" (UID: "1f0d04e9-a9a4-4913-9350-00ce30b70788"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:56:07.489201 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.489136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0d04e9-a9a4-4913-9350-00ce30b70788-kube-api-access-5n48k" (OuterVolumeSpecName: "kube-api-access-5n48k") pod "1f0d04e9-a9a4-4913-9350-00ce30b70788" (UID: "1f0d04e9-a9a4-4913-9350-00ce30b70788"). InnerVolumeSpecName "kube-api-access-5n48k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:56:07.587907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.587864    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d04e9-a9a4-4913-9350-00ce30b70788-openshift-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:56:07.587907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.587897    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-maas-api-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:56:07.587907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.587907    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d04e9-a9a4-4913-9350-00ce30b70788-metrics-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:56:07.587907 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:07.587917    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n48k\" (UniqueName: \"kubernetes.io/projected/1f0d04e9-a9a4-4913-9350-00ce30b70788-kube-api-access-5n48k\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\""
Sep 15 09:56:08.070505 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.070460    2579 generic.go:356] "Generic (PLEG): container finished" podID="1f0d04e9-a9a4-4913-9350-00ce30b70788" containerID="9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f" exitCode=0
Sep 15 09:56:08.070691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.070509    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"
Sep 15 09:56:08.070691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.070547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48" event={"ID":"1f0d04e9-a9a4-4913-9350-00ce30b70788","Type":"ContainerDied","Data":"9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f"}
Sep 15 09:56:08.070691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.070603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48" event={"ID":"1f0d04e9-a9a4-4913-9350-00ce30b70788","Type":"ContainerDied","Data":"a8b3aea485d7c2e63eaf7c01d2993a340d004e2b64429650c2e2461865a4fb71"}
Sep 15 09:56:08.070691 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.070628    2579 scope.go:117] "RemoveContainer" containerID="9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f"
Sep 15 09:56:08.081514 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.081491    2579 scope.go:117] "RemoveContainer" containerID="9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f"
Sep 15 09:56:08.081873 ip-10-0-140-130 kubenswrapper[2579]: E0915 09:56:08.081848    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f\": container with ID starting with 9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f not found: ID does not exist" containerID="9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f"
Sep 15 09:56:08.081918 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.081888    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f"} err="failed to get container status \"9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f\": rpc error: code = NotFound desc = could not find container \"9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f\": container with ID starting with 9a0ddacbab468e2869940db7bb2ba5a53133f54a1b14a7fc387d949e98989a4f not found: ID does not exist"
Sep 15 09:56:08.094061 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.094020    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"]
Sep 15 09:56:08.098980 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.098947    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-68fc5fb999-7zz48"]
Sep 15 09:56:08.675294 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:08.675259    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f0d04e9-a9a4-4913-9350-00ce30b70788" path="/var/lib/kubelet/pods/1f0d04e9-a9a4-4913-9350-00ce30b70788/volumes"
Sep 15 09:56:12.797614 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:12.797574    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:56:21.095668 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:21.095626    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:56:27.704054 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:27.704017    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8czjn"]
Sep 15 09:56:36.939742 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:36.939707    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-767c4c8b76-mbwh7_0d3fc6aa-0b4e-4401-a35a-876bf7c445f0/copy-manifests/0.log"
Sep 15 09:56:36.954137 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:36.954102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-767c4c8b76-mbwh7_0d3fc6aa-0b4e-4401-a35a-876bf7c445f0/manager/0.log"
Sep 15 09:56:37.413594 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:37.413553    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c4664ccd7-6q5sg_71e9da57-40b3-4057-9be7-204256ef9acc/maas-api/0.log"
Sep 15 09:56:38.896211 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:38.896179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-5zn66_a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2/manager/0.log"
Sep 15 09:56:39.007684 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:39.007656    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-srwxw_98d6af91-4a3f-4737-bbfa-6d7832b93b6f/kuadrant-console-plugin/0.log"
Sep 15 09:56:39.250762 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:39.250727    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8czjn_1ac1449c-8d70-47ac-bd7c-a802cb83deb3/limitador/0.log"
Sep 15 09:56:40.058948 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:40.058871    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6df85cfc68-5tsp2_11f27408-50b5-4cc0-8de5-514f9cd191a2/kube-auth-proxy/0.log"
Sep 15 09:56:40.872388 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:40.872322    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4jv_aeecbd62-6c59-4f8f-bc6e-0e01c65bcd03/main/0.log"
Sep 15 09:56:41.461136 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:41.461073    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-xckhd_0cc002ec-ba8f-42a4-ae54-43337c4ac31a/main/0.log"
Sep 15 09:56:41.568923 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:41.568890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5lfvd_28c3ebc8-4517-4d4b-938b-dba6596d36bb/main/0.log"
Sep 15 09:56:50.220328 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:50.220272    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-szh7l_92876d96-e3e2-4b10-ad5d-067e6ea18d22/global-pull-secret-syncer/0.log"
Sep 15 09:56:50.300536 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:50.300488    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g6sck_b725cd06-0dd2-44be-b821-1017c1c994e9/konnectivity-agent/0.log"
Sep 15 09:56:50.377522 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:50.377484    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-130.ec2.internal_c12dd1aab63b90cc211915d100e750d9/haproxy/0.log"
Sep 15 09:56:54.628641 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:54.628605    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-5zn66_a3b9e2e1-fffb-4c6b-96da-2bb9ffda68b2/manager/0.log"
Sep 15 09:56:54.658453 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:54.658431    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-srwxw_98d6af91-4a3f-4737-bbfa-6d7832b93b6f/kuadrant-console-plugin/0.log"
Sep 15 09:56:54.736619 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:54.736596    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8czjn_1ac1449c-8d70-47ac-bd7c-a802cb83deb3/limitador/0.log"
Sep 15 09:56:56.602119 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:56.602037    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-w2hdj_e55de041-5230-4a5e-a871-ef60649a3c40/kube-state-metrics/0.log"
Sep 15 09:56:56.629743 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:56.629722    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-w2hdj_e55de041-5230-4a5e-a871-ef60649a3c40/kube-rbac-proxy-main/0.log"
Sep 15 09:56:56.655753 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:56.655735    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-w2hdj_e55de041-5230-4a5e-a871-ef60649a3c40/kube-rbac-proxy-self/0.log"
Sep 15 09:56:56.683057 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:56.683029    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-747c78868f-bjv5v_98e2482a-d61f-4d97-a53e-ea27b1960c4d/metrics-server/0.log"
Sep 15 09:56:56.748042 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:56.747991    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86n9r_d72ca623-0b27-4c15-ba21-5f3b6f0c7505/node-exporter/0.log"
Sep 15 09:56:56.771587 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:56.771566    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86n9r_d72ca623-0b27-4c15-ba21-5f3b6f0c7505/kube-rbac-proxy/0.log"
Sep 15 09:56:56.797210 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:56.797193    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86n9r_d72ca623-0b27-4c15-ba21-5f3b6f0c7505/init-textfile/0.log"
Sep 15 09:56:57.009522 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.009501    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-5bdjz_1fa2f9ec-477d-4607-a9ae-492efc929777/kube-rbac-proxy-main/0.log"
Sep 15 09:56:57.035548 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.035525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-5bdjz_1fa2f9ec-477d-4607-a9ae-492efc929777/kube-rbac-proxy-self/0.log"
Sep 15 09:56:57.061123 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.061101    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-5bdjz_1fa2f9ec-477d-4607-a9ae-492efc929777/openshift-state-metrics/0.log"
Sep 15 09:56:57.117775 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.117719    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71a8d178-6e82-498d-8929-1bd0f4e3ddb8/prometheus/0.log"
Sep 15 09:56:57.153149 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.153128    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71a8d178-6e82-498d-8929-1bd0f4e3ddb8/config-reloader/0.log"
Sep 15 09:56:57.181275 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.181258    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71a8d178-6e82-498d-8929-1bd0f4e3ddb8/thanos-sidecar/0.log"
Sep 15 09:56:57.211058 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.211041    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71a8d178-6e82-498d-8929-1bd0f4e3ddb8/kube-rbac-proxy-web/0.log"
Sep 15 09:56:57.244706 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.244688    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71a8d178-6e82-498d-8929-1bd0f4e3ddb8/kube-rbac-proxy/0.log"
Sep 15 09:56:57.275833 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.275817    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71a8d178-6e82-498d-8929-1bd0f4e3ddb8/kube-rbac-proxy-thanos/0.log"
Sep 15 09:56:57.322154 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.322138    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71a8d178-6e82-498d-8929-1bd0f4e3ddb8/init-config-reloader/0.log"
Sep 15 09:56:57.363977 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.363956    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-hbdtf_acb00edc-c2af-4c5f-9da5-86d74e899cc5/prometheus-operator/0.log"
Sep 15 09:56:57.392696 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.392644    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-hbdtf_acb00edc-c2af-4c5f-9da5-86d74e899cc5/kube-rbac-proxy/0.log"
Sep 15 09:56:57.420069 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.420053    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-sdv9z_f64a93cc-ed79-48bf-bb1f-7b1bed808b83/prometheus-operator-admission-webhook/0.log"
Sep 15 09:56:57.613610 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.613589    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-586844fb88-nhqkz_321eb189-0735-4e62-9ddc-4975d8a3c1d9/thanos-query/0.log"
Sep 15 09:56:57.647553 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.647500    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-586844fb88-nhqkz_321eb189-0735-4e62-9ddc-4975d8a3c1d9/kube-rbac-proxy-web/0.log"
Sep 15 09:56:57.695922 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.695903    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-586844fb88-nhqkz_321eb189-0735-4e62-9ddc-4975d8a3c1d9/kube-rbac-proxy/0.log"
Sep 15 09:56:57.722522 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.722498    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-586844fb88-nhqkz_321eb189-0735-4e62-9ddc-4975d8a3c1d9/prom-label-proxy/0.log"
Sep 15 09:56:57.868055 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.868033    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-586844fb88-nhqkz_321eb189-0735-4e62-9ddc-4975d8a3c1d9/kube-rbac-proxy-rules/0.log"
Sep 15 09:56:57.923042 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:57.923021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-586844fb88-nhqkz_321eb189-0735-4e62-9ddc-4975d8a3c1d9/kube-rbac-proxy-metrics/0.log"
Sep 15 09:56:59.427010 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:56:59.426982    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-s5wgg_12ac97ae-6fdc-479b-9f49-1b3bec6717a9/networking-console-plugin/0.log"
Sep 15 09:57:00.590900 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.590864    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"]
Sep 15 09:57:00.591267 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.591254    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f0d04e9-a9a4-4913-9350-00ce30b70788" containerName="maas-api"
Sep 15 09:57:00.591312 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.591268    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f0d04e9-a9a4-4913-9350-00ce30b70788" containerName="maas-api"
Sep 15 09:57:00.591384 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.591371    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f0d04e9-a9a4-4913-9350-00ce30b70788" containerName="maas-api"
Sep 15 09:57:00.595874 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.595844    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.598545 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.598522    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tznnh\"/\"kube-root-ca.crt\""
Sep 15 09:57:00.598683 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.598666    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tznnh\"/\"openshift-service-ca.crt\""
Sep 15 09:57:00.599375 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.599354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tznnh\"/\"default-dockercfg-7ll79\""
Sep 15 09:57:00.603798 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.603772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"]
Sep 15 09:57:00.667643 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.667619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-lib-modules\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.667742 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.667658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-proc\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.667742 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.667732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-sys\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.667816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.667756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-podres\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.667816 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.667786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28ktx\" (UniqueName: \"kubernetes.io/projected/a5eb7dda-8cc8-488d-9237-dfd56faf5775-kube-api-access-28ktx\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.768316 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.768290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-sys\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.768433 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.768324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-podres\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.768433 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.768406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-sys\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.768510 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.768436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-podres\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.768510 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.768443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28ktx\" (UniqueName: \"kubernetes.io/projected/a5eb7dda-8cc8-488d-9237-dfd56faf5775-kube-api-access-28ktx\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.768510 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.768466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-lib-modules\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.768510 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.768493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-proc\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.768667 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.768574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-lib-modules\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.768667 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.768580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5eb7dda-8cc8-488d-9237-dfd56faf5775-proc\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.777445 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.777421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28ktx\" (UniqueName: \"kubernetes.io/projected/a5eb7dda-8cc8-488d-9237-dfd56faf5775-kube-api-access-28ktx\") pod \"perf-node-gather-daemonset-wncbf\" (UID: \"a5eb7dda-8cc8-488d-9237-dfd56faf5775\") " pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:00.906902 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:00.906883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:01.182253 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:01.182177    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"]
Sep 15 09:57:01.184950 ip-10-0-140-130 kubenswrapper[2579]: W0915 09:57:01.184921    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda5eb7dda_8cc8_488d_9237_dfd56faf5775.slice/crio-591a9a22879a2f35fd338331198c9fb7eea451f5f9a678a67c1c2c377952ac47 WatchSource:0}: Error finding container 591a9a22879a2f35fd338331198c9fb7eea451f5f9a678a67c1c2c377952ac47: Status 404 returned error can't find the container with id 591a9a22879a2f35fd338331198c9fb7eea451f5f9a678a67c1c2c377952ac47
Sep 15 09:57:01.330713 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:01.330681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf" event={"ID":"a5eb7dda-8cc8-488d-9237-dfd56faf5775","Type":"ContainerStarted","Data":"af4c845a530d4e57467ceaf1bc829be62565bc6906d02ba0792b1a12b810bb81"}
Sep 15 09:57:01.330862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:01.330730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf" event={"ID":"a5eb7dda-8cc8-488d-9237-dfd56faf5775","Type":"ContainerStarted","Data":"591a9a22879a2f35fd338331198c9fb7eea451f5f9a678a67c1c2c377952ac47"}
Sep 15 09:57:01.330862 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:01.330817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:01.351529 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:01.351487    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf" podStartSLOduration=1.3514750979999999 podStartE2EDuration="1.351475098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:57:01.349121554 +0000 UTC m=+2585.178913457" watchObservedRunningTime="2026-09-15 09:57:01.351475098 +0000 UTC m=+2585.181267000"
Sep 15 09:57:02.567793 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:02.567763    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pt8lk_a8d07861-7136-4a13-93a8-d959c6632229/dns/0.log"
Sep 15 09:57:02.587353 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:02.587314    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pt8lk_a8d07861-7136-4a13-93a8-d959c6632229/kube-rbac-proxy/0.log"
Sep 15 09:57:02.701499 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:02.701464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2vzlg_6b6c0984-533c-4e9e-a633-28abecd84a99/dns-node-resolver/0.log"
Sep 15 09:57:03.394081 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:03.394058    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vjcp6_f65720d5-a59e-4e28-a582-c4ae0b861511/node-ca/0.log"
Sep 15 09:57:04.885201 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:04.885171    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6df85cfc68-5tsp2_11f27408-50b5-4cc0-8de5-514f9cd191a2/kube-auth-proxy/0.log"
Sep 15 09:57:06.041218 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:06.041186    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vz4v2_f3c8d2b7-0f7a-42fd-b51a-29062eed16a5/serve-healthcheck-canary/0.log"
Sep 15 09:57:06.551793 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:06.551767    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7vlm_6260a04b-ea51-4bde-9d0a-797236eb14e5/kube-rbac-proxy/0.log"
Sep 15 09:57:06.578443 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:06.578421    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7vlm_6260a04b-ea51-4bde-9d0a-797236eb14e5/exporter/0.log"
Sep 15 09:57:06.610450 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:06.610431    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7vlm_6260a04b-ea51-4bde-9d0a-797236eb14e5/extractor/0.log"
Sep 15 09:57:07.346167 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:07.346136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tznnh/perf-node-gather-daemonset-wncbf"
Sep 15 09:57:09.622652 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:09.622618    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-767c4c8b76-mbwh7_0d3fc6aa-0b4e-4401-a35a-876bf7c445f0/manager/0.log"
Sep 15 09:57:09.641450 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:09.641417    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-767c4c8b76-mbwh7_0d3fc6aa-0b4e-4401-a35a-876bf7c445f0/copy-manifests/0.log"
Sep 15 09:57:11.139764 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:11.139731    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f69b6c8dc-nmrxg_35884fd4-cd31-4d87-880e-c17679682226/manager/0.log"
Sep 15 09:57:11.526614 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:11.526587    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6df85cfc68-5tsp2_11f27408-50b5-4cc0-8de5-514f9cd191a2/kube-auth-proxy/0.log"
Sep 15 09:57:13.575517 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:13.575486    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6df85cfc68-5tsp2_11f27408-50b5-4cc0-8de5-514f9cd191a2/kube-auth-proxy/0.log"
Sep 15 09:57:16.182915 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:16.182874    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-9nwb5_1bc192c2-9130-4b77-bb0f-5aa33c269203/migrator/0.log"
Sep 15 09:57:16.202030 ip-10-0-140-130 kubenswrapper[2579]: I0915 09:57:16.202002    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-9nwb5_1bc192c2-9130-4b77-bb0f-5aa33c269203/graceful-termination/0.log"